diff --git a/docs/versioned_docs/version-2.12.x/introduction/releases.md b/docs/versioned_docs/version-2.12.x/introduction/releases.md index f52d14cd8..ac93730f9 100644 --- a/docs/versioned_docs/version-2.12.x/introduction/releases.md +++ b/docs/versioned_docs/version-2.12.x/introduction/releases.md @@ -175,15 +175,15 @@ The latest release versions of each of the engine are as follows: - Kubernetes 1.18 or higher release is recommended as this release contains the following updates that will not be compatible with older Kubernetes releases. - The CSI components have been upgraded to: - - k8s.gcr.io/sig-storage/csi-attacher:v3.1.0 - - k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0 - - k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0 - - k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1 (for Mayastor CSI volumes) - - k8s.gcr.io/sig-storage/csi-resizer:v1.1.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0 - - k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) - - k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/csi-attacher:v3.1.0 + - registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.1.0 + - registry.k8s.io/sig-storage/csi-provisioner:v2.1.0 + - registry.k8s.io/sig-storage/csi-provisioner:v2.1.1 (for Mayastor CSI volumes) + - registry.k8s.io/sig-storage/csi-resizer:v1.1.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v4.0.0 + - registry.k8s.io/sig-storage/snapshot-controller:v4.0.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) - If you are upgrading from a version of cStor operators older than 2.6 to this version, you will need to manually delete the cStor CSI driver object prior to upgrading. `kubectl delete csidriver cstor.csi.openebs.io`. For complete details on how to upgrade your cStor operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools. @@ -235,15 +235,15 @@ Here are some of the key highlights in this release. - Kubernetes 1.17 or higher release is recommended as this release contains the following updates that will not be compatible with older Kubernetes releases. - The CSI components have been upgraded to: - - k8s.gcr.io/sig-storage/csi-attacher:v3.1.0 - - k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0 - - k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0 - - k8s.gcr.io/sig-storage/csi-provisioner:v2.1.1 (for Mayastor CSI volumes) - - k8s.gcr.io/sig-storage/csi-resizer:v1.1.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0 - - k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) - - k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/csi-attacher:v3.1.0 + - registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.1.0 + - registry.k8s.io/sig-storage/csi-provisioner:v2.1.0 + - registry.k8s.io/sig-storage/csi-provisioner:v2.1.1 (for Mayastor CSI volumes) + - registry.k8s.io/sig-storage/csi-resizer:v1.1.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v4.0.0 + - registry.k8s.io/sig-storage/snapshot-controller:v4.0.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) - If you are upgrading from a version of cStor operators older than 2.6 to this version, you will need to manually delete the cStor CSI driver object prior to upgrading. `kubectl delete csidriver cstor.csi.openebs.io`. For complete details on how to upgrade your cStor operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools. - The CRD API version has been updated for the cStor custom resources to v1. If you are upgrading via the helm chart, you might have to make sure that the new CRDs are updated. https://github.com/openebs/cstor-operators/tree/master/deploy/helm/charts/crds - The e2e pipelines include upgrade testing only from 1.6 and higher releases to 2.7. If you are running on release older than 1.6, OpenEBS recommends you upgrade to the latest version as soon as possible. @@ -301,14 +301,14 @@ Here are some of the key highlights in this release. - Kubernetes 1.17 or higher release is recommended as this release contains the following updates that will not be compatible with older Kubernetes releases. - The CSI components have been upgraded to: - - k8s.gcr.io/sig-storage/csi-attacher:v3.1.0 - - k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0 - - k8s.gcr.io/sig-storage/csi-provisioner:v2.1.0 - - k8s.gcr.io/sig-storage/csi-resizer:v1.1.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v4.0.0 - - k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) - - k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/csi-attacher:v3.1.0 + - registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.1.0 + - registry.k8s.io/sig-storage/csi-provisioner:v2.1.0 + - registry.k8s.io/sig-storage/csi-resizer:v1.1.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v4.0.0 + - registry.k8s.io/sig-storage/snapshot-controller:v4.0.0 + - registry.k8s.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) - If you are upgrading from an older version of cStor operators to this version, you will need to manually delete the cStor CSI driver object prior to upgrading. `kubectl delete csidriver cstor.csi.openebs.io`. For complete details on how to upgrade your cStor operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools. @@ -370,8 +370,8 @@ Here are some of the key highlights in this release. - quay.io/k8scsi/csi-snapshotter:v4.0.0 - quay.io/k8scsi/csi-resizer:v1.1.0 - quay.io/k8scsi/csi-attacher:v3.1.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) - - k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) - If you are upgrading from an older version of cStor operators to this version, you will need to manually delete the cstor CSI driver object prior to upgrade. `kubectl delete csidriver cstor.csi.openebs.io`. For complete details on how to upgrade your cStor operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools. @@ -498,8 +498,8 @@ Here are some of the key highlights in this release. - quay.io/k8scsi/csi-snapshotter:v4.0.0 - quay.io/k8scsi/csi-resizer:v1.1.0 - quay.io/k8scsi/csi-attacher:v3.1.0 - - k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) - - k8s.gcr.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/csi-snapshotter:v3.0.3 (for cStor CSI volumes) + - registry.k8s.io/sig-storage/snapshot-controller:v3.0.3 (for cStor CSI volumes) - If you are upgrading from an older version of cStor Operators to this version, you will need to manually delete the cstor CSI driver object prior to upgrade. `kubectl delete csidriver cstor.csi.openebs.io`. For complete details on how to upgrade your cStor Operators, see https://github.com/openebs/upgrade/blob/master/docs/upgrade.md#cspc-pools. diff --git a/website/src/blogs/a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver.md b/website/src/blogs/a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver.md index f7a83510f..ff4130d18 100644 --- a/website/src/blogs/a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver.md +++ b/website/src/blogs/a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver.md @@ -152,7 +152,7 @@ In this exercise we will use the OpenEBS LVM CSI driver though it will work with spec: containers: - name: nginx - image: k8s.gcr.io/nginx-slim:0.8 + image: registry.k8s.io/nginx-slim:0.8 ports: - containerPort: 80 name: web diff --git a/website/src/posts.json b/website/src/posts.json index 359ce7950..e75161f43 100644 --- a/website/src/posts.json +++ b/website/src/posts.json @@ -1 +1 @@ -[{"id":1,"title":"Mayastor 1.0 has arrived","author":"Prasoon Pushkar","author_info":"Prasoon works as a Product Marketing Manager in DataCore Software. His day to day activities revolve around ideating & executing positioning & messaging to grow OpenEBS & Mayastor. His free time is all about travelling, munching & playing FIFA","date":"18-01-2022","tags":["Mayastor","Kubernetes","Solutions"],"excerpt":"In this blog, we will go through Mayastor architecture, evolution of project over the years, benefits for the community and different use cases","content":"\nMayastor was started in late 2019 as a sub-project of OpenEBS and has been under development by the [MayaData](https://mayadata.io/) team as an advancement of earlier storage engines. Over the years, Mayastor remained in beta and the culmination of efforts of the engineering team of MayaData, the Cloud Native Computing Foundation and the vibrant open-source community led us to build the foundation of various features and stability. In that spirit, today we are proud to release the first community version of Mayastor . \n\nIn this blog, we will go through Mayastor architecture, evolution of project over the years, benefits for the community and different use cases.\n\n## What is Mayastor \n\nAs a sub-project of the [OpenEBS](https://openebs.io/) [CAS](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/) (Container Attached Storage) solution, the Mayastor storage engine is designed to provide persistent storage that is easy to manage and readily deployable. Mayastor is a cloud-native storage platform by MayaData that abstracts storage resources to enable persistent storage with low abstraction overhead for stateful Kubernetes applications. \n\nBuilt on the Container Attached Storage framework, the storage engine leverages a hyper-converged deployment model, where data is stored locally then shared with other nodes via synchronous replication.\n\n## Mayastor Architecture \n\nThe platform is built for cloud-native storage orchestration and is workload-driven to simplify Kubernetes storage for enterprises. Through a declarative data plane, Mayastor abstracts storage resources to enable persistent storage for Kubernetes applications. To do so, Mayastor storage engine components are deployed as containers in Kubernetes, allowing for simple scaling, provisioning and management of storage for clusters.\n\n**Control Plane -** The Mayastor control plane is primarily a single instance Kubernetes controller that implements both the Container Storage Interface (CSI) specification and private interfaces for the storage system. The controller is containerized and runs as deployment in Kubernetes. The control plane also includes CSI plugins deployed on each node to implement CSI protocol services. \n\n**Data Plane -** The Mayastor data plane creates a Nexus for every Persistent Volume Claim (PVC) loosely coupled to the storage class. Each Nexus instance acts as a virtual storage router, connecting the PVC with a Persistent Volume (PV). The Nexus instance provides an abstraction of the input-output controller while managing all requests for the Persistent Volume attached to the PVC. \n\n**NVMe and NVMe-oF-** Mayastor is designed to leverage the benefits of both NVMe and NVMe-oF to support low latency workloads for converged and segregated storage. With Mayastor, organizations can benefit from the performance capabilities of new generation high performance SSDs.\n\n**[Storage Performance Development Kit (SPDK)](https://spdk.io/) :** Developed and maintained by Intel, SPDK offers valuable libraries, tools and user-level I/O execution to enhance the performance of storage media in a data centre. Mayastor integrates with the SPDK, helping storage administrators leverage its features for improved throughput performance and efficient SSD storage. The platform also helps provision high availability storage using poll mode drivers and user-level I/O execution capabilities. \n\n## Why Choose Mayastor ? \n\nThe Mayastor data plane leverages the full performance potential of storage systems, delivering high IOPS with less than 10% overhead. The declarative data plane utilizes developments of NVMe and SPDK to enable easy storage management. The following section outlines various features and benefits Mayastor offers. \n\n## Benefits of Mayastor 1.0 \n\n**Reduced Latency – Meet IO-intensive Applications Requirements**\n\nMayastor is developed using an architecture that is targeted at meeting the storage performance requirements of IO-intensive Kubernetes applications. Mayastor uses a CSI driver with application and platform awareness to speed up migration for single and replicated volumes. \n\n**Guaranteed Efficient Memory Transfer** \n\nThe Mayastor data plane is implemented in Rust, which includes a safe compiler that guarantees efficient memory transfer. NVMe architecture also ensures that each SSD controller communicates with one ring per CPU. This ensures the I/O controllers aren’t prone to internal locking, enabling simpler memory management. \n\n**Cloud-Agnostic Storage – Eliminate Vendor Lock-in** \n\nMayastor storage controllers are orchestrated by Kubernetes, and data is accessed via containers. As the storage engine is deployed using a CSI driver, volumes can be provisioned using any container orchestrator or runtime, eliminating vendor lock-in. This construct allows organizations to use Mayastor to orchestrate any open-source storage service running on different Kubernetes distributions. \n\n**Flexible Horizontal Scalability** \n\nMayastor integrates with OpenEBS Logical Volume Management (LVM), which pools underlying volumes and divides them into immutable logical units. These units can be replicated and distributed across multiple availability zones, enabling flexible horizontal scalability for storage workloads. \n\n**Seamless storage replication** \n\nThe Mayadata storage controller simplifies the provisioning and replication of storage instances across pods, thereby enabling efficient deployment of cross-cloud storage for stateful applications. \n \n\n**Highly Available persistent data storage** \n\nMayastor relies on Intel’s industry-leading Storage Performance Development Kit to deliver reliable access to persistent storage services and devices. The storage engine offers resilient volume services through synchronous replication that guarantees availability. \n \n\n**Easy deployment and management of storage services** \n\nThe gRPC API simplifies operations management while providing security and easy configuration of the Mayastor data plane. The control plane implements application-aware data placement for fine-grained control over timeouts, restarts and errors of Kubernetes storage volumes. \n \n\n**Low-overhead storage abstraction** \n\nMayastor follows the Container-Attached Storage (CAS) pattern, which orchestrates storage as a workload in Kubernetes. With CAS, storage scales predictably, allowing workloads to operate seamlessly with the storage system. Besides this, as Mayastor is built from the ground up to leverage the benefits of NVMe storage, the platform further reduces performance overhead. \n\n## Mayastor Evolution \n\nThe section below explores Mayastor’s release history, noting the major milestones in the storage engine’s development. \n\n**V 0.1.0** \n\nThis Alpha release was intended mostly for testing and development purposes and featured persistent volumes that could be exported over ISCSI transport. \n\n**V 0.2.0** \n\nThe first pre-release version of Mayastor included various useful features, such as: \n\n- NVMe-oF Support - The storage engine had the capability to create and share a Nexus with an established front-end target. \n\n- Mayastor Node CRD - With the resource definition enabled, storage admins can configure observability and persistence for Mayastor nodes. \n\n- Rebuild Process - This mechanism allowed to synchronously bring in new replicas without disrupting IO workloads by scheduling concurrent rebuild tasks. \n \n\n**V 0.3.0** \n\nReleased on 12th August 2020, the second pre-release version of the storage engine included features such as: \n\n- NVMe-oF support for CSI plugins \n\n- Block mode for raw block access \n\n- Direct device access for Mayastor pools \n\n- Automatic replica replacement \n\n**V 0.4.0** \n\nReleased on September 4, 2020, the fourth major version included numerous enhancements and new features. Major enhancements included: \n\n- Restart tolerance for the rebuild process \n\n- Nexus I/O internal retries for failed operations \n\n- Automatic generation of Mayastor pool names \n \n\nNew features included: \n\n- Mayastor support for Kubernetes clusters built on K3s \n\n- Enhanced user documentation with simple navigation, improved UI and search \n \n\n**V 0.5.0** \n\nFeatures and enhancements of Mayastor 0.5.0 include: \n\n- Block device discovery through the gRPC API \n\n- Storage pool finalizers to prevent inadvertent deletion of volumes that contain replicas \n \n\nMayastor V0.6.0 and 0.7.0 were maintenance releases, while V0.8.0 was a patch release that included various stability fixes in preparation for the first major production release.\n\n\n**Mayastor 1.0** \n\nMayastor 1.0 is released now with multiple enhancements and stable features that make the storage engine respond more resilient to failures. Some features to be included in the release are: \n\n- TCP access support \n\n- N-way synchronous replication for workload protection \n\n- Support for Prometheus metrics and visualization in Grafana \n\n- CSI driver for volume management \n\n\n## Mayastor Use-Cases \n\n**Storage unification** \n\nThe Mayastor data plane can be deployed on any cloud running a Kubernetes environment. Storage teams can deploy storage controllers on multiple environments, on-premises and from cloud storage providers, and manage them from a central interface. \n \n\n**Low latency workloads** \n\nNVMe and NVMe-oF architectures reduce storage network latency and support high-speed storage media by eliminating bottlenecks caused by SATA connections. \n \n\n**Programmatic storage access** \n\nWith CAS, the storage controller does not have to make system calls for IOPS operations. Storage access is simplified since the controllers write to file and block storage devices directly from within the application. \n \n\n**Server consolidation** \n\nMayastor can manage heterogeneous workloads by orchestrating multiple data planes from a single control plane. This allows storage experts to reduce the number of compute resources required to process IO requests, reducing storage costs while increasing efficiency. \n \n\n**Containers on Micro-VM servers** \n\nMayastor volumes can be used to provide storage over vhost users, enabling storage teams to persist data for multi-tenant containers in virtual machines. Mayastor storage controllers can be deployed on lightweight virtual machines (MicroVMs) and function-based services to enable the efficient execution of IO tasks for serverless and container workloads. \n\n## Summary \n\nOpenEBS Mayastor is the first Container Attached Storage engine developed with the tremendous performance capabilities of the NVMe protocol itself in mind. Written in Rust and open source, Mayastor should be the preferred choice for workloads that need the ease of use and portability of OpenEBS plus the performance otherwise only delivered by the much harder to manage and to protect direct access to high performing disks and cloud volumes. \n \n\nTo know more on how Mayastor can help provision lightning fast storage solution for your dynamic workloads, drop us a message [here](https://openebs.io/community). Alternatively, you can also visit the official [documentation](https://mayastor.gitbook.io/introduction/quickstart/deploy-a-test-application) to learn more about Mayastor, its appropriate use-cases, and feature announcements. \n","notHasFeatureImage":false,"slug":"mayastor-1.0-has-arrived"},{"id":2,"title":"Install WordPress using OpenEBS Dynamic NFS Provisioner","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"02-11-2021","tags":["OpenEBS NFS","NFS Provisioner","RWX Volume Provisioning","Dynamic NFS Provisioner"],"excerpt":"Guide to deploy WordPress by using OpenEBS dynamic NFS Provisioner","content":"\n### Introduction\n In large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. Mayadata, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. This blog is mainly for users who require ReadWriteMany(RWX) for Kubernetes applications.\n\n## OpenEBS Dynamic NFS Provisioner\n OpenEBS Dynamic NFS Provisioner will enable you to dynamically provision ReadWriteMany(RWX/MultiNode ReaderWriter) volume on any type of Kubernetes Persistent Volumes which supports dynamic provisioning, RWX is achieved by launching NFS Server on the Persistent Volume. In simple words, dynamic-nfs-provisioner will expose ReadWriteOnce volume as ReadWriteMany volume.\n\n**Terminology:**\n\n ![NFS terminology](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-nfs-terminology.png)\n\n Few workloads in Kubernetes require ReadWriteMany type of volumes, for example, WordPress, Magento, etc. In this blog, I will walk through the steps to configure WordPress with a dynamic-nfs-provisioner.\n\n\n## Deploy WordPress by using dynamic NFS provisioner\n For deploying WordPress it is required to have dynamic NFS provisioner installed, So let’s first install dynamic NFS provisioner and then WordPress application.\n\n### Prerequisites for deploying dynamic NFS provisioner\n- Minimum Kuberentes Version 1.18\n- NFS Client must be installed on all nodes where the application pod schedules.\n For Ubuntu/Debian run below command to install nfs client:\n ```sh\n sudo apt install nfs-common\n ```\n- Storage provider(supports dynamic provisioning) to serve backend volume claim requests. In this blog, we are going to use OpenEBS localpv-hostpath as a backend storage provider.\n\n### Install NFS Provisioner\n NFS provisioner will dynamically provision NFS volumes, install NFS provisioner using below helm commands and it will deploy NFS provisioner with basic configurations, NFS provisioner also accepts [values.yaml](https://github.com/openebs/dynamic-nfs-provisioner/tree/develop/deploy/helm/charts#configuration)\n\n - To install NFS provisioner add repository and then install nfs-provisioner\n\n #### Add OpenEBS repo for installation\n - Use the below command to add helm repo for OpenEBS\n ```sh\n helm repo add openebs https://openebs.github.io/charts\n ```\n - Once repo has been added successfully, update helm repo using the following command:\n ```sh\n helm repo update\n ```\n\n #### Installing OpenEBS NFS Provisioner\n Once the helm repo addition is successful, run the `helm install` command as specified below, in this command, we are disabling the local-pv provisioner which doesn’t require for NFS Provisioner to work and configure __.nfsStorageClass.backendStorageClass__ corresponding to backend StorageProvider\n\n ```sh\n helm install openebs openebs/openebs -n openebs --create-namespace \\\n --set legacy.enabled=false \\\n --set ndm.enabled=false \\\n --set ndmOperator.enabled=false \\\n --set localProvisioner.enabled=false \\\n --set nfs-provisioner.enabled=true \\\n --set nfs-provisioner.nfsStorageClass.backendStorageClass=\n ```\n - After successful helm installation, you can view the openebs-nfs-provisioner and openebs-kernel-nfs StorageClass will get deployed.\n\n **Output**:\n\n ```sh\n [develop@develop ~]$ kubectl get po -n openebs\n NAME READY STATUS RESTARTS AGE\n openebs-nfs-provisioner-79b6ccd59-v8p6s 1/1 Running 0 4m12s\n\n [develop@develop~]$ kubectl get sc\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\n openebs-kernel-nfs openebs.io/nfsrwx Delete Immediate false 67s\n ```\n\n **Optional(installation of local-pv-provisioner)**\n\n The below command will install OpenEBS local-pv provisioner along with NFS provisioner, as stated above local-pv is used as backend StorageProvider in this blog\n\n ```sh\n helm install openebs openebs/openebs -n openebs --create-namespace \\\n --set legacy.enabled=false \\\n --set ndm.enabled=false \\\n --set ndmOperator.enabled=false \\\n --set localProvisioner.enabled=true \\\n --set localProvisioner.hostpathClass.name=openebs-hostpath \\\n --set localprovisioner.basePath=/var/openebs/hostpath \\\n --set nfs-provisioner.enabled=true \\\n --set nfs-provisioner.nfsStorageClass.backendStorageClass=openebs-hostpath\n ```\n\n **Note**: `localprovisioner.basePath` defines the custom hostpath directory to provide storage for backend PVCs.\n\n### Install WordPress\n\n [WordPress](https://en.wikipedia.org/wiki/WordPress) is one of the most famous blogging platforms and also it is bloggers first choice because of it's easy to handle nature. One can dynamically scale web servers of WordPress(content management systems [CMS]) by making use of Horizontal Pod Autoscaler feature of Kubernetes i.e increasing load will automatically scale the wordpress pods, users can install WordPress in multiple ways, but in this blog, we are going to install from its Helm repository by using [MariaDB](https://en.wikipedia.org/wiki/MariaDB) server to satisfy the requirements of WordPress server.\n\n #### Add helm repo for WordPress\n\n - Use the below command to add helm repo for WordPress\n ```sh\n helm repo add bitnami https://charts.bitnami.com/bitnami\n ```\n - Once repo has been added successfully, update helm repo using the following command\n ```sh\n helm repo update\n ```\n\n #### Installing WordPress\n Once the repo has been updated successfully, we can install WordPress referring to dynamic NFS provisioner storageclass. In this case storageclass will be openebs-kernel-nfs, if you have different storageclass then update the value of --set persistence.storageClass accordingly. WordPress also accepts various other parameters, for more information click [here](https://github.com/bitnami/charts/tree/master/bitnami/wordpress#parameters)\n\n ```sh\n helm install my-release -n wordpress --create-namespace \\\n --set wordpressUsername=admin \\\n --set wordpressPassword=password \\\n --set mariadb.auth.rootPassword=secretpassword \\\n --set mariadb.primary.persistence.enabled=true \\\n --set mariadb.primary.persistence.storageClass=openebs-hostpath \\\n --set mariadb.primary.persistence.accessModes={ReadWriteOnce} \\\n --set persistence.storageClass=openebs-kernel-nfs \\\n --set persistence.accessModes={ReadWriteMany} \\\n --set volumePermissions.enabled=true \\\n --set autoscaling.enabled=true \\\n --set autoscaling.minReplicas=2 \\\n --set autoscaling.maxReplicas=6 \\\n --set autoscaling.targetCPU=80 \\\n bitnami/wordpress\n ```\n\n Above helm installation should create following resources in the cluster:\n\n - Two WordPress application pod instances with RWX persistent volume.\n - One Mariadb application pod instance with RWO persistent volume.\n\n Results of helm wordpress installation:\n ```sh\n kubectl get po -n wordpress\n NAME READY STATUS RESTARTS AGE\n my-release-mariadb-0 1/1 Running 0 2m34s\n my-release-wordpress-85dc688bb9-24jx2 1/1 Running 0 2m34s\n my-release-wordpress-85dc688bb9-wkfpd 1/1 Running 1 2m18s\n ```\n\n Once the installation process is succeeded, follow the steps mentioned in output of _helm install_ command to access WordPress from your browser:\n\n ![WordPress login page](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-login-page.png)\n\n Now it's time to hack WordPress:\n\n ![WordPress landing page](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-landing-page.png)\n\n**Deployment View**:\n\n ![WordPress Deployment view](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-deployment-view.jpg)\n\n\n **Background details on how NFS provisioner on provisioning RWX volume:**\n\n Once the PersistentVolumeClaim is created by referring to NFS provisioner storageclass, NFS provisioner will process the volume request and perform following actions:\n - Create Backend PVC and wait till the backend storage provider provides a volume.\n - Create NFS server deployment on backend PVC to expose RWO volumes as RWX volume via kernel NFS service.\n - Create NFS server kubernetes service which helps for other services to communicate with NFS server.\n - Create PersistentVolume(PV) for requested PersistentVolumeClaim.\n\n View of resources created after provisioning volume:\n\n ```sh\n kubectl get all -n openebs\n\n NAME READY STATUS RESTARTS AGE\n pod/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323-57645d68f8-hrvvm 1/1 Running 0 12m\n \n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n service/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 ClusterIP 10.100.164.113 2049/TCP,111/TCP 12m\n \n NAME READY UP-TO-DATE AVAILABLE AGE\n deployment.apps/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 1/1 1 1 12m\n \n kubectl get pvc -n openebs\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 Bound pvc-4bde2312-15a6-4b06-a309-069b47ce569c 10Gi RWO openebs-hostpath 14m\n ```\n\n### Benefits of using dynamic NFS Provisioner\n Dynamic NFS provisioner has various advantages, few of them are:\n - Provides configurable options to reduce recovery times(i.e maintaining HighAvailability of server)\n - **LeaseTime**: Lease time defines the renewal period(in seconds) for client connection.\n - **GraceTime**: Grace time defines the recovery period(in seconds) to reclaim locks.\n\n Recovery options can be configured via StorageClass annotations:\n ```yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-rwx\n annotations:\n openebs.io/cas-type: nfsrwx\n cas.openebs.io/config: |\n # LeaseTime defines the renewal period(in seconds) for client state\n - name: LeaseTime\n value: 30\n # GraceTime defines the recovery period(in seconds) to reclaim locks\n - name: GraceTime\n value: 30\n ...\n ...\n ...\n ```\n - Provides options to launch an NFS server on a specific set of nodes by using [node-affinity](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/tutorial/node-affinity.md#nfs-server-node-affinity) feature.\n - Supports [manual](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/tutorial/nfs-volume-resize.md) expansion of NFS(RWX) volume, for automation please track issue [#37](https://github.com/openebs/dynamic-nfs-provisioner/issues/37).\n - Provides an option to make NFS volume available only for [specific set of applications](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/troubleshooting/non-root-application-accesing-nfs-volume.md#intro) which has matching permissions.\n","notHasFeatureImage":true,"slug":"install-wordpress-using-openebs-dynamic-nfs-provisioner"},{"id":3,"title":"OpenEBS 3.0 Release","author":"Kiran Mova","author_info":"Founder, Contributor and Maintainer of OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-09-2021","tags":["OpenEBS"],"excerpt":"Announcing OpenEBS 3.0 release - with ability to create 9 types of volumes. ","content":" \nWe released OpenEBS in early 2017 as one of the first solutions to adopt the Container Attached Storage pattern to deliver per workload storage in part by building upon Kubernetes. OpenEBS is now one of the most popular storage solutions for Kubernetes; our metrics indicate that usage of OpenEBS has increased significantly in the last three years, by approximately 7x, with the most popular workloads being those that are resilient including NoSQL DBs that often do not use shared storage. \n\nA lot of the growth in usage by OpenEBS is due to the overall growth in Kubernetes for stateful workloads. The [CNCF’s 2020 Survey](https://www.cncf.io/wp-content/uploads/2020/11/CNCF_Survey_Report_2020.pdf) highlighted 55% of surveyed organizations now use stateful applications in production. **The survey also highlighted MayaData / OpenEBS is in the top-5 list of popular storage solutions.**\n\nOver the years, OpenEBS has steadily evolved into a full stack storage software with swappable storage engines, related control plane components, and integrations into other cloud native projects. \n\nA quick summary:\n- **OpenEBS 1.0** was launched with the goal of simplifying the journey of users, who were onboarding Stateful workloads into Kubernetes. The initial release included: \n * A stable distributed block storage called “Jiva” based on the Longhorn engine. \n * An initial version of the CStor engine based on ZFS that was much more resilient to node failures and optimized for working with Hard Drives and RAID. \n * Also components like Node Disk Manager that helped to manage local storage on Kubernetes nodes and Dynamic LocalPV Provisioners.\n- **OpenEBS 2.0** focused on the stability of the data plane and building of an automated test framework, and the introduction of additional engines based on user feedback. This release included: \n * Making OpenEBS a CNCF project - which meant streamlining a lot of project governance, development processes and moving towards CNCF provided tools. \n * CStor data plane was declared stable, with support for incremental backup/restore support via Velero. \n * E2e tests for all of the OpenEBS Engines using the Litmus framework, which went on to become a project of its own.\n * ZFS LocalPV provisioners (CSI Driver) was introduced.\n * Initial release of Mayastor - which was the result of investigation into inherent performance limitations of earlier approaches including Jiva and CStor.\n- **OpenEBS 3.0** has been about enhancing the usability and stability aspects of all the previously introduced engines as well as support for new engines. The release is also about supporting the newer Kubernetes releases. This release includes:\n * CSI Drivers for CStor and Jiva and tools to migrate volumes to the newer CSI drivers. The legacy provisioners are deprecated with 3.0.0 and users are required to migrate to the corresponding CSI drivers asap. \n * Several enhancements to existing LocalPV flavors and introduction of new types of LocalPV. \n * A new control plane for Mayastor is being worked on that is better designed to handle scale and resiliency.\n * Also included are the initial version of a kubectl plugin for OpenEBS and a prometheus grafana mixin for managing the OpenEBS storage and volumes. \n\n## What’s new in OpenEBS 3.0?\n\nOpenEBS 3.0 (See [Release Notes](https://github.com/openebs/openebs/releases/tag/v3.0.0)) is a culmination of efforts geared towards laying the foundation for making it easier to onboard and accept community contributions, making each of the engine operators ready for future Kubernetes releases, making it easy to manage and troubleshoot various engines. This has been achieved via migration to latest Kubernetes constructs, ease of use improvements, bug fixes and most importantly refactoring the control plane and e2e test suites to independently enhance and release each of the engines. \n\nI highlight a few of these enhancements below. \n\nOpenEBS includes various storage engines developed for different workloads and use-cases. OpenEBS data engines can be classified into two categories - Local and Replicated.\n\n### LocalPV - New Features and Enhancements\n\nOpenEBS uses LocalPV provisioners to connect applications directly with storage from a single node. This storage object, known as LocalPV, is subject to the availability of the node on which it is mounted, making it a handy feature for fault-tolerant applications who prefer local storage over traditional shared storage. The OpenEBS LocalPV provisioner enables Kubernetes-based stateful applications to leverage several types of local storage features ranging from raw block devices to using capabilities of filesystems on top of those devices like LVM and ZFS. \n\nOpenEBS 3.0 includes the following enhancement to the LocalPV provisioner:\n* Support for 3 new types of LocalPVs namely LVM LocalPV, Rawfile LocalPV, Partition Local PV in addition to the previously supported Hostpath LocalPV, Device LocalPV and ZFS LocalPV. \n* [OpenEBS Hostpath LocalPV (stable)](https://github.com/openebs/dynamic-localpv-provisioner), the first and the most widely used LocalPV now supports enforcing XFS quota, ability to use a custom node label for node affinity (instead of the default `kubernetes.io/hostname`) \n* [OpenEBS ZFS LocalPV (stable)](https://github.com/openebs/zfs-localpv), used widely for production workloads that need direct and resilient storage has added new capabilities like:\n * Velero plugin to perform incremental backups that make use of the copy-on-write ZFS snapshots.\n * CSI Capacity based scheduling used with `waitForFirstConsumer` bound Persistent Volumes. \n * Improvements to inbuilt volume scheduler (used with `immediate` bound Persistent Volumes) that can now take into account the capacity and the count of volumes provisioned per node.\n* [OpenEBS LVM LocalPV (stable)](https://github.com/openebs/lvm-localpv), can be used to provision volume on top of LVM Volume Groups and supports the following features:\n * Thick (Default) or Thin Provisioned Volumes \n * CSI Capacity based scheduling used with `waitForFirstConsumer` bound Persistent Volumes. \n * Snapshot that translates into LVM Snapshots \n * Ability to set QoS on the containers using LVM Volumes.\n * Also supports other CSI capabilities like volume expansion, raw or filesystem mode, metrics. \n* [OpenEBS Rawfile LocalPV (beta)](https://github.com/openebs/rawfile-localpv), is a preferred choice for creating local volumes using a sparse file within a sub-directory that supports capacity enforcement, filesystem or block volumes.\n* [OpenEBS Device LocalPV (beta)](https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md#provisioning-localpv-device-persistent-volume), is a preferred choice for running workloads that have typically worked well with consuming the full disks in block mode. This provisioner uses NDM to select the block device. \n* [OpenEBS Partition LocalPV (an alpha engine)](https://github.com/openebs/device-localpv), is under active development and is being deployed in select users for creating volumes by dynamically partitioning a disk with the requested capacity from the PVC. \n\n### ReplicatedPV - New Features and Enhancements\n\nOpenEBS uses ReplicatedPV provisioners to connect applications to volumes - whose data is synchronously replicated to multiple storage nodes. This storage object, known as ReplicatedPV, is highly available and can be mounted from multiple nodes in the clusters. OpenEBS supports three types of ReplicatedPVs Jiva (based on Longhorn and iSCSI), CStor (based on ZFS and iSCSI) and Mayastor (based on SPDK and NVMe). \n\nSome enhancements to replicated storage engines in OpenEBS 3.0 include:\n* [OpenEBS Jiva (stable)](https://github.com/openebs/jiva-operator), has added support for a CSI Driver and Jiva operator that include features like:\n * Enhanced management of the replicas \n * Ability to auto-remount the volumes marked as read-only due to iSCSI time to read-write. \n * Faster detection of the node failure and helping Kubernetes to move the application out of the failed node to a new node. \n * 3.0 also deprecates the older Jiva volume provisioners - that was based on the kubernetes external storage provisioner. There will be no more features added to the older provisioners and users are requested to migrate their Volumes to CSI Drivers as soon as possible. \n* [OpenEBS CStor (stable)](https://github.com/openebs/cstor-operators), has added support for a CSI Driver and also improved customer resources and operators for managing the lifecycle of CStor Pools. This 3.0 version of the CStor includes:\n * The improved schema allows users to declaratively run operations like replacing the disks in mirrored CStor pools, add new disks, scale-up replicas, or move the CStor Pools to a new node. The new custom resource for configuring CStor is called CStorPoolCluster (CSPC) compared to older StoragePoolCluster(SPC). \n * Ability to auto-remount the volumes marked as read-only due to iSCSI time to read-write. \n * Faster detection of the node failure and helping Kubernetes to move the application out of the failed node to a new node. \n * 3.0 also deprecates the older CStor volume provisioners and pool operators based on SPC - that was based on the kubernetes external storage provisioner. There will be no more features added to the older provisioners and users are requested to migrate their Pools to CSPC and Volumes to CSI Drivers as soon as possible.\n* [OpenEBS Mayastor (beta)](https://github.com/openebs/mayastor), is under active development and currently is building a new and enhanced control plane to manage the mayastor pools and volumes. In the current release, the changes to the Mayastor include:\n * Support for deprecating the MOAC based control plane in favor of the new control plane. \n * Enhanced control plane to handle node failure scenarios and move the volumes to new nodes. \n * Stabilizing the Mayastor data engine for durability and performance. \n * Enhanced E2e tests.\n\n### Other Notable Features and Enhancements\n\nBeyond the improvements to the data engines and their corresponding control plane, there are several new enhancements that will help with ease of use of OpenEBS engines:\n* Several fixes and enhancements to the Node Disk Manager like automatically adding a reservation tag to devices, detecting filesystem changes and updating the block device CR (without the need for a reboot), metrics exporter and an API service that can be extended in the future to implement storage pooling or cleanup hooks. \n* [Dynamic NFS Provisioner](https://github.com/openebs/dynamic-nfs-provisioner) that allows users to launch a new NFS server on any RWO volume (called backend volume) and expose an RWX volume that saves the data to the backend volume. \n* Kubernetes Operator for automatically upgrading Jiva and CStor volumes that are driven by a Kubernetes Job \n* Kubernetes Operator for automatically migrating CStor Pools and Volumes from older pool schema and legacy (external storage based) provisioners to the new Pool Schema and CSI volumes respectively. \n* [OpenEBS CLI (a kubectl plugin)](https://github.com/openebs/openebsctl) for easily checking the status of the block devices, pools (storage) and volumes (PVs). \n* [OpenEBS Dashboard](https://github.com/openebs/monitoring) (a prometheus and grafana mixin) that can be installed via jsonnet or helm chart with a set of default Grafana Dashboards and AlertManager rules for OpenEBS storage engines. \n* Enhanced OpenEBS helm chart that can easily enable or disable a data engine of choice. The 3.0 helm chart stops installing the legacy CStor and Jiva provisioners. If you would like to continue to use them, you have to set the flag “legacy.enabled=true”. \n* OpenEBS helm chart includes sample kyverno policies that can be used as an option for PodSecurityPolicies(PSP) replacement. \n* OpenEBS images are delivered as multi-arch images with support for AMD64 and ARM64 and hosted on DockerHub, Quay and GHCR. \n* Support for installation in air gapped environments. \n* Enhanced Documentation and Troubleshooting guides for each of the engines located in the respective engine repositories. \n* A new and improved design for the [OpenEBS website](https://openebs.io/). \n\n### Deprecated Features\n\nAs announced in [June earlier this year](https://github.com/openebs/openebs/releases/tag/v2.10.0), the non-csi provisioners for CStor and Jiva are deprecated with 3.0. You can still continue to use them till Dec 2021. The older provisioners are released with the v2.12.2 version at the moment and only patch releases (to fix severe security vulnerabilities) will be supported going forward. \n\n## What is Next after OpenEBS 3.0? \n\nWe originally developed OpenEBS that focused on solving the core issue of Kubernetes storage - that is the lack of support to manage stateful applications, especially modern resilient workloads that do not use shared storage and prefer the per workload approach of Container Attached Storage. That remains our focus today - and as mentioned innovation seems to be accelerating as the community of users and contributors grows at an accelerated rate. \nI am especially thankful to the OpenEBS users and organizations that have taken the time to fill out [Adopters.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md) to share how they are using OpenEBS, and how it is solving their common problems while leveraging Kubernetes for data. \n\nI will also be talking about the new features, upgrades & bug fixes and giving glimpses into what is coming in OpenEBS 3.1 and 4.0 at the CNCF On-Demand Webinar: OpenEBS 3.0 : What’s in it on September 30th. Feel free to register & attend: [https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-openebs-30-whats-in-it/](https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-openebs-30-whats-in-it/)\n\nTo learn more about recent updates, developer documentation, and scheduled releases, please feel free to use any of the following resources.\n- [https://github.com/openebs/openebs/releases](https://github.com/openebs/openebs/releases)\n- [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md)\n- [https://openebs.io/docs](https://openebs.io/docs)\n- [https://openebs.io/community](https://openebs.io/community)\n\n\n\n\n\n\n\n\n\n\n\n\n\n","notHasFeatureImage":true,"slug":"openebs-3.0-release"},{"id":4,"title":"A hands on demo of Volume Populator using OpenEBS LVM CSI driver","author":"Shovan Maity","author_info":"Shovan works as a Software Engineer at MayaData, who's experienced in Load Balancer, gRPC, WebSocket, REST APIs, and has good hands-on experience on Kubernetes. In his free time, Shovan likes to read blogs on distributed systems. He also likes Travelling and Photography.","date":"05-07-2021","tags":["OpenEBS","LVM CSI driver"],"excerpt":"In this blog we will write a volume populator and create a volume with that populator. This is written using lib-volume-populator. This is a control loop on PVC that maintains the lifecycle of the PV and PVC for volume populator.","content":"\nIn this blog we will write a volume populator and create a volume with that populator. Source code of this volume populator is available [here](https://github.com/shovanmaity/s3-populator). This is written using [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator). This is a control loop on PVC that maintains the lifecycle of the PV and PVC for volume populator. If we want to write a volume populator we need to write a CRD spec like [this](https://github.com/shovanmaity/s3-populator/blob/main/types.go) and we need to code on how to write the data on the volume. We can build or reuse any plugin that can write data from source to the volume. Here we will use [s3 sync](https://github.com/shovanmaity/s3-sync) image to write data from s3 to local volume. This can copy data from a bucket or a particular directory in a bucket. This may have some other use cases. In this example we will create, build a react app, push it to MinIo and then we will use that s3 bucket to create volumes for nginx instances.\n\nIn this exercise we will use the OpenEBS LVM CSI driver though it will work with any CSI driver and we will perform this demo in a local minikube cluster. The steps are given below. \n\n* Create a minikube cluster with AnyVolumeDataSource alpha feature enabled mode.\n \n ```\n minikube start --feature-gates=AnyVolumeDataSource=true\n ```\n\n* Install OpenEBS LVM CSI driver.\n \n ```\n kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/lvm-operator.yaml\n ```\n\n* Create PV and VG in the host machine.\n \n ```\n truncate -s 10G /tmp/disk.img\n sudo losetup -f /tmp/disk.img --show\n #get loop device id and replace X with it\n sudo pvcreate /dev/loopX\n sudo vgcreate lvmvg /dev/loopX\n ```\n\n* Create a storage class to create lvm volumes using this volume group.\n\n ``` \n kubectl create -f - << EOF\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-lvmpv\n parameters:\n storage: lvm\n volgroup: lvmvg\n provisioner: local.csi.openebs.io\n EOF\n ```\n\n* Install volume populator controller.\n\n ``` \n kubectl apply -f https://raw.githubusercontent.com/shovanmaity/s3-populator/main/crd.yaml\n kubectl apply -f https://raw.githubusercontent.com/shovanmaity/s3-populator/main/deploy.yaml\n ```\n\n* Run MinIO on the local machine as a S3 source.\n\n ```\n docker run -p 9000:9000 \\\n -e \"MINIO_ROOT_USER=minioadmin\" \\\n -e \"MINIO_ROOT_PASSWORD=minioadmin\" \\\n minio/minio server /data\n ```\n\n* Create and build a react app in local machine.\n \n ```\n npx create-react-app my-app\n cd my-app\n npm build\n ```\n\n* Create a S3 bucket to push the static files.\n \n ```\n AWS_ACCESS_KEY_ID=minioadmin \\\n AWS_SECRET_ACCESS_KEY=minioadmin \\\n AWS_REGION=us-east-1 \\\n aws --endpoint-url http://192.168.0.190:9000 s3api create-bucket --bucket my-bucket\n ```\n\n* Make sure we are inside the build folder of the react app and then push the static files to s3.\n \n ```\n AWS_ACCESS_KEY_ID=minioadmin \\\n AWS_SECRET_ACCESS_KEY=minioadmin \\\n AWS_REGION=us-east-1 \\\n aws --endpoint-url http://192.168.0.190:9000 s3 cp . s3://my-bucket --recursive\n ```\n\n* Create a S3 populator cr with S3 and bucket related information.\n\n ```\n kubectl create -f - << EOF\n apiVersion: example.io/v1\n kind: S3Populator\n metadata:\n name: s3-populator-1\n namespace: default\n spec:\n url: http://192.168.0.190:9000\n id: minioadmin\n secret: minioadmin\n region: us-east-1\n bucket: my-bucket\n key: /\n EOF\n ```\n\n* Create a nginx service to access it.\n \n ```\n kubectl create -f - << EOF\n apiVersion: v1\n kind: Service\n metadata:\n name: nginx\n namespace: default\n labels:\n app: nginx\n spec:\n ports:\n - port: 80\n name: web\n type: NodePort\n selector:\n app: nginx\n EOF\n ```\n\n* Create a nginx sts.\n \n ```\n kubectl create -f - << EOF\n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: web\n namespace: default\n spec:\n serviceName: nginx\n replicas: 1\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: k8s.gcr.io/nginx-slim:0.8\n ports:\n - containerPort: 80\n name: web\n volumeMounts:\n - name: www\n mountPath: /usr/share/nginx/html\n volumeClaimTemplates:\n - metadata:\n name: www\n spec:\n dataSource:\n apiGroup: example.io\n kind: S3Populator\n name: s3-populator-1\n storageClassName: \"openebs-lvmpv\"\n accessModes: [ \"ReadWriteOnce\" ]\n resources:\n requests:\n storage: 500Mi\n EOF\n ```\n\n* Get the service IP and port and open it in any browser. We will be able to see the same content(react app) in our nginx app.\n\nPlease join our community if you have any feedback or queries on the above demo and OpenEBS.","notHasFeatureImage":false,"slug":"a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver"},{"id":5,"title":"Life cycle of PVC with Volume Populator","author":"Shovan Maity","author_info":"Shovan works as a Software Engineer at MayaData, who's experienced in Load Balancer, gRPC, WebSocket, REST APIs, and has good hands-on experience on Kubernetes. In his free time, Shovan likes to read blogs on distributed systems. He also likes Travelling and Photography.","date":"05-07-2021","tags":["PVC","Volume Populator"],"excerpt":"This blog is on the life cycle of pvc with a volume populator. Volume populator feature is in alpha in Kubernetes (v1.21 release). Creating a volume with base or seed data is one of the requirements in storage.","content":"\nThis blog is on the life cycle of pvc with a [volume populator](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators). Volume populator feature is in alpha in Kubernetes (v1.21 release). Creating a volume with base or seed data is one of the requirements in storage. In Kubernetes 1.12, the _DataSource_ field is added to the PVC spec. The field was implemented as a _TypedLocalObjectReference_ to give flexibility in the future about what objects could be data sources for new volumes. Since then, it supports only two things to be the source of a new volume - existing PVC(clone volume) and snapshots(restore a snapshot). Implementation of these two data sources relies on the CSI plugin to do the actual work. As it can point to any CR in Kubernetes validation on the DataSource field of PVC is relaxed and relies on a new async controller to perform the validation on those data sources and provide feedback to users. A new CRD is introduced to register valid dataSource, and we need to create a resource for individual volume populators to inform that the volume populator is a valid dataSource for a PVC.\n\nIn the first part we will see the volume lifecycle without a volume populator. In this process involved components are _**API server**, **etcd**, **CSI external provisioner**_ and _**CSI controller plugin**._ We will not go into the CSI node plugin, we will stop as soon as PVC is in bound state. Let's understand the responsibility of some modules present here.\n\n![volume lifecycle without a volume populator](https://lh6.googleusercontent.com/rX9n7NSjDPiwL0xf1YR0hfOF0pf5TY7qgzMrKhhQyARxREFOUakXS0aAWFZ4Y1MKE7b6tUZ33iYVidZdPyOXPhgW0G60lmIGZOz5KadBJaQirYK6pjTWUix2Hvk4laXXzFPfg803)\n\n**CSI external provisioner**\n\nThis is a CSI driver sidecar component. It is a control loop on PVC. As clone and restore volumes come under CSI workflow if _DataSource_ is present and it is PVC or snapshot then only it takes some actions for that PVC else it will ignore that PVC.\n\n**CSI controller plugin**\n\nThis is a CSI driver sidecar component provided by the storage provider(SP). It is a gRPC server which implements CSI controller services. CSI external provisioner makes a CreateVolume gRPC call to it once new PVC is created.\n\n* User creates a pvc and it is written in etcd with the help of API server and the pvc goes to the pending state.\n* CSI external provisioner observes that pvc and makes _CreateVolume_ call to the CSI controller plugin sidecar. On successful response it creates a PV for that PVC and applies bi-directional reference between them.\n* Once bi-directional reference is done, pvc goes to bound state.\n\nNow we will see the volume lifecycle with the data populator. Involved components are _**API server**, **etcd**, **CSI external provisioner**, **CSI controller plugin** and **Volume Populator controller**._ We will not go into the CSI node plugin, we will stop as soon as PVC is in bound state. Let's understand the responsibility of some modules present here. \n\n![volume lifecycle with the data populator](https://lh4.googleusercontent.com/-uuZ8z1w6y-jGSOomfn_VuIQt_6nOtOUv11fS1Qac0p7FhneZMM617gfjf49ek-YKiKVHpEYqbc2787UhIvxq7lTeVm4OKCsMFwdy_iZ-fNpzrgqaU4RP2864LNlcTC0JWKCeXoF)\n\n**Volume Populator Controller**\n\nIt is a control loop on PVC, responsible to manage PVC pointing to a particular dataSource. Namespace in which this controller is deployed is reserved for creating some intermediate PVC and pod to populate the data in the volume. If we create a PVC in that namespace for our application with a valid dataSource it will not work, this control loop will skip that PVC. This controller adds a finalizer to the initial PVC. It is written using [lib volume populator](https://github.com/kubernetes-csi/lib-volume-populator). [This](https://github.com/kubernetes-csi/lib-volume-populator/tree/master/example/hello-populator) is an example of a data populator written using lib volume populator. One new CRD _Volume Populator_ is introduced_._ Like CSI drivers we have to register the volume populator.\n\n* User creates a PVC with dataSource pointing to a volume populator and it is written in etcd with the help of API server and the PVC goes to the pending state.\n* CSI external provisioner observes that PVC and takes no action as the dataSource is not pointing to a PVC or snapshot.\n* Volume populator controller is another watcher on pvc. It creates a PVC’ which is the same as the PVC created by the user without the dataSource field.\n* As PVC’ does not have a dataSource CSI external provisioner makes a CreateVolume gRPC call to the CSI controller plugin. Once the gRPC call is successful PV’ is created and bi-directional reference applied on PVC’ and PV’. Then PVC’ goes to the bound state.\n* Once PVC’ is in bound state, the volume populator controller creates a pod to populate the data for that PVC’.\n* Once data is populated it patches PVC and PV’ and applies a bi-directional reference between  them. After this PVC will be in bound state and PVC’ will be in lost state. Then the populator pod and PVC’ are deleted by this controller.\n\nAs mentioned above in core Kubernetes validation on the DataSource field is relaxed now we can create PVC pointing to any data source. To add validation around it one new asynchronous PVC [controller](https://github.com/kubernetes-csi/volume-data-source-validator) is added. This controller is responsible for posting warning events if PVC is pointing to invalid DataSource. One new CRD of kind VolumePopulator and API version populator.storage.k8s.io/v1alpha1 is introduced to specify a valid volume populator.\n\nHere is one valid data source -\n\n```\n kind: VolumePopulator\n apiVersion: populator.storage.k8s.io/v1alpha1\n metadata:\n name: valid-populator\n sourceKind:\n group: valid.storage.k8s.io\n kind: Valid\n```\n\nHere is an example PVC with valid data source -\n\n```\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: valid-pvc-datasource\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 1Gi\n dataSource:\n apiGroup: valid.storage.k8s.io\n kind: Valid\n name: valid\n```\n\nHere is an example of PVC with invalid data source -\n\n```\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: invalid-pvc-datasource\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 1Gi\n dataSource:\n apiGroup: invalid.storage.k8s.io\n kind: Invalid\n name: invalid\n```\n\nEvent in invalid data source PVC-\n\n```\n shovan@probot:~$ kubectl get event | grep Warning\n 58s Warning UnrecognizedDataSourceKind persistentvolumeclaim/invalid-pvc-datasource The data source for this PVC does not match any registered VolumePopulator\n\n shovan@probot:~$ kubectl describe pvc invalid-pvc-datasource\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal ExternalProvisioning 115s (x2 over 115s) persistentvolume-controller waiting for a volume to be created, either by external provisioner \"k8s.io/minikube-hostpath\" or manually created by system administrator\n Normal Provisioning 114s k8s.io/minikube-hostpath_minikube_e7c97d0c-eb52-42bc-af55-d8a167ab3987 External provisioner is provisioning volume for claim \"default/invalid-pvc-datasource\"\n Normal ProvisioningSucceeded 113s k8s.io/minikube-hostpath_minikube_e7c97d0c-eb52-42bc-af55-d8a167ab3987 Successfully provisioned volume pvc-7df410d5-1efe-4d69-b481-57cbde9f10cd\n Warning UnrecognizedDataSourceKind 33s (x6 over 114s) data-source-validator The data source for this PVC does not match any registered VolumePopulator\n```\n\nPlease join our [community](https://openebs.io/community) if you have any feedback or queries on the article or anything related to [OpenEBS](https://openebs.io/)\n","notHasFeatureImage":false,"slug":"life-cycle-of-pvc-with-volume-populator"},{"id":6,"title":"Logical Volume Management (LVM) on Kubernetes with OpenEBS LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"17-06-2021","tags":["Kubernetes","OpenEBS","LocalPV","LVM"],"excerpt":"In large production environments, various physical disks are typically pooled to handle user and application storage needs. Logical Volume Management (LVM) is a storage virtualization technology that is a part of Linux which is widely used as a way to help administrators and developers abstract the physical configuration of storage devices for flexible and simpler provisioning. ","content":"\n## Introduction to Logical Volume Management\n\nIn large production environments, various physical disks are typically pooled to handle user and application storage needs. Logical Volume Management (LVM) is a **storage virtualization** technology that is a part of Linux which is widely used as a way to help administrators and developers **abstract the physical configuration** of storage devices for flexible and simpler provisioning. The LVM tool gathers all storage devices into volume groups, creates logical units depending on application needs, and then allocates them to particular groups. By doing so, LVM gives greater control, flexibility and easier abstraction of physical storage enabling use cases such as the replacement of local devices, local RAID and the ability to manage the sharing of JBODs across multiple hosts.\n\nOpenEBS offers a **LVM option for LocalPV** that allows Kubernetes administrators to dynamically provision Persistent Volumes. This article delves into the architecture of Logical Volume Management and how to set up and use the **OpenEBS LVM LocalPV** to provision and manage storage for a Kubernetes cluster.\n\n## LVM Architecture & Concepts\n\nAt its foundation, LVM combines several individual physical disks into one Volume Group that can be subdivided into multiple Logical Volumes. This is achieved by creating several layered abstractions between the physical drives and the end-user’s storage partitions.\n\n![A typical PVC user request flow](/images/blog/a-typical-pvc-user-request-flow.png) \n(***A typical PVC user request flow***)\n\nThe various storage management structure that LVM uses to simplify the management of conflicting end-user storage needs, in increasing order of virtualization are: \n\n## Physical Volumes\n\nThese are regular storage devices that include physical disks such as HDDs, SSDs and RAIDs. Such units are essentially used as basic raw devices for achieving a level of abstraction. To manage a physical disk, the LVM assigns a header to it which it uses while allocating volumes.\n\n### Volume Groups (VG)\n\nVolume Groups are LVM’s higher level of abstraction and consist of one or more physical volumes. Once a physical storage device is added into a VG, the files stored inside it are partitioned into multiple smaller logical **extents** as specified by administrators when creating the Volume Group. Besides partition size, administrators can also set a VG’s security permissions and other specifications.\n\n### Logical Volumes\n\nA Volume Group (VG) is partitioned into several Logical Volumes (LVs) that can be perceived as flexible partitions of volume. LVs are the primary abstract interface through which end-users and applications interact with storage. \n\nVolumes in a Volume Group are partitioned into smaller segments known as **Extents**. While Physical volumes are divided into **Physical Extents**, Logical Volumes are divided into **Logical Extents**. An extent size is the smallest amount of space that the LVM can allocate since the Logical Volumes are just abstract mapping between physical and logical extents.\n\nExtents give LVM flexibility since logical extents do not need to be directly and continuously mapped to physical extents. An LVM can duplicate and rearrange the physical extents in a physical volume without changing logical extents, eliminating the need to migrate users. This gives an efficient way of resizing logical extents by adding or removing physical extents from the volume.\n\n### OpenEBS LVM LocalPV\n\nThe OpenEBS Container Storage Interface (CSI) driver allows Kubernetes administrators to dynamically resize Local Persistent Volumes using Logical Volume Management. There are several capabilities of OpenEBS LocalPV LVM that readers are encouraged to learn. These are available on both OpenEBS docs or via the published [OpenSource roadmap](https://github.com/orgs/openebs/projects/30). OpenEBS LocalPV LVM is a widely deployed project, which is under active development with the assistance of the broader OpenEBS community, including the hyper-scale eCommerce firm Flipkart. In the section below, we cover the steps to install a OpenEBS LVM driver in a Kubernetes cluster, along with highlighting its possible use-cases.\n\n## Installing OpenEBS LVM into a Kubernetes Cluster\n\n### Setup\n\nFor the CSI LVM driver to work on a cluster, it must meet the following prerequisites:\n\n1. All nodes should have LVM2 utilities installed on their host systems. Additional details on LVM2 utilities can be found here: \n\n[Ubuntu - http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html](http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html)\n\n[Red Hat - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_logical_volumes/index](http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html).\n\n2. A Volume Group of block storage devices has been set up to provision volume for the cluster. \n\n3. The user has administrative rights so they can install RBAC components into the default namespace for objects created by the Kubernetes System: **kube-system**.\n\nThe OpenEBS CSI LVM driver supports Kubernetes 1.18 and newer versions, the Ubuntu OS and runs on LVM2 utilities. \n\nIf you have a real physical device, you can use the command accordingly or can use a loopback device for testing. The following commands are used to locate the disk to be managed by LVM.\n\n $ truncate -s 1024G /tmp/disk.img //marks the extents\n $ sudo losetup -f /tmp/disk.img --show //finds unused device\n\nThe Volume Group is created on all nodes using commands similar to the following:\n\n $ sudo pvcreate /dev/sdb\n $ sudo vgcreate lvmvg /dev/sdb\n\nThis creates the Volume Group `lvmvg`. The same process must be followed to create Volume Groups on all nodes. \n\n### Installation\n\nThe OpenEBS CSI LVM driver can then be installed from the official repository by running the command:\n\n $ kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/lvm-operator.yaml\n\nThe following command can be used to check whether the driver and its components have been installed: \n\n $ kubectl get pods -n kube-system -l role=openebs-lvm\n\nOnce the driver has been successfully installed, administrators can now use it to provision volumes through the steps to follow below.\n\n### Provisioning Volumes with the LVM Driver\n\nProvisioning Volumes with the OpenEBS LVM driver consists of three steps:\n\n1. Creating a storage class\n2. Creating a Persistent Volume Claim (PVC)\n3. Deploying the Application\n\nThe manifest file for a storage class that allows the LVM driver to create volumes on specific nodes is as shown:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: darwin-lvmpv\n allowVolumeExpansion: true\n parameters:\n volgroup: \"lvmvg\"\n provisioner: local.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: kubernetes.io/hostname\n values:\n - lvmpv-node1\n - lvmpv-node2\n\nThe YAML file above instructs the LVM to deploy volumes from the Volume Group `lvmvg` to nodes `lvmpv-node1` and `lvmpv-node2`.\n\nThe Persistent Volume Claim for this storage class will look similar to:\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: pvc-lvmpv\n spec:\n storageClassName: darwin-lvmpv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nThe application can now be allocated storage resources from the `darwin-vg` Volume Group.\n\n## OpenEBS LocalPV LVM Use-Cases\n\nOnce the LVM driver has been installed in a cluster, it allows dynamic provisioning of LocalPV Volumes to Kubernetes applications. This is useful for applications such as:\n\n* Dynamic resizing of assigned physical and logical volumes\n* Creating smaller storage environments for tests and upgrades\n* Tenant Isolation\n* Backup and Recovery using redundant partitions \n\nBy simplifying the management of users’ conflicting storage needs by abstracting logical and physical volumes, Logical Volume Management (LVM) is critical for managing production workloads efficiently. To do so, OpenEBS through a Container Storage Interface (CSI) Driver allows Kubernetes administrators to dynamically resize Local Persistent Volumes using Logical Volume Management. This enables an application-level **storage virtualization layer** for **greater control**, **flexibility** and **easier abstraction** of physical storage.","notHasFeatureImage":true,"slug":"logical-volume-management-(lvm)-on-kubernetes-with-openebs-localpv"},{"id":7,"title":"Provisioning OpenEBS Jiva volumes via CSI","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"09-06-2021","tags":["CSI","OpenEBS","Jiva"],"excerpt":"Container Storage Interface (CSI), is the new model for integrating storage system drivers with container orchestration systems like Kubernetes. This new interface is a major benefit to the container ecosystem as it standardizes the model of integrating storage systems with container orchestration systems.","content":"\n[Container Storage Interface (CSI)](https://github.com/container-storage-interface/spec), is the new model for integrating storage system drivers with container orchestration systems like [Kubernetes](https://kubernetes.io/). This new interface is a major benefit to the container ecosystem as it standardizes the model of integrating storage systems with container orchestration systems. Specifically for Kubernetes, it frees the storage system driver from being tied to the Kubernetes release schedule due to it being incorporated in the same code base. With CSI, storage system drivers can now be installed asynchronously to container orchestration releases, providing faster bug fixes and features.\n\nThe Openebs Jiva CSI driver provides a CSI interface that allows it to manage the lifecycle of OpenEBS Jiva volumes for persistent volumes. OpenEBS has recently released a beta version of the [Jiva CSI driver](https://github.com/openebs/jiva-operator) for one of its Container Attached Storage engines - [Jiva](https://github.com/openebs/jiva).\n\nAs of 2.7.0, the supported features are volume provisioning, de-provisioning, volume resize, raw block volume, and volume usage metrics. There are other key differences between the external-provisioner and CSI implementations, are benefits to users, like\n\n- Auto remount\n- Protecting against nodes mounting an empty directory accidentally and re-writing. \n- Fix for multi-attach errors on onprem k8s clusters\n- Standardised with CSI and further enhancements coming in K8s\n\nThis blog will demonstrate how to configure and consume Jiva volumes using the CSI Driver. \n\n## Prerequisites\n\n1. Kubernetes version 1.18 or higher\n2. iSCSI initiator utils installed on all the worker nodes\n3. You have access to install RBAC components into the openebs namespace.\n4. For resize feature, verify ExpandCSIVolumes and ExpandInUsePersistentVolumes feature gates are enabled on the kubelet of all worker nodes and kube-apiserver.\n\n## Installation\n\n### Using Jiva operator: \n\n1. Install openebs local PV provisioner to create hostpath based volumes.\n\n```\nkubectl apply -f https://openebs.github.io/charts/openebs-operator-lite.yaml\n```\n\n2. Install the latest Jiva-operator release.\n\n```\n$ kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml\n```\n\n3. Using Jiva Helm chart:\n\n```\n$ helm repo add openebs-jiva https://openebs.github.io/jiva-operator\n$ helm repo update \n$ helm install openebs-jiva/jiva\n```\n\nMore info related to Jiva charts, its configurable parameters, and their default values can be found [here](https://github.com/openebs/jiva-operator/tree/master/deploy/helm/charts).\n\n## Configure Jiva CSI Driver\n\n### OpenEBS Jiva CSI driver comprises of 2 components:\n\n1. A controller component launched as a StatefulSet, implementing the CSI controller services. The Control Plane services are responsible for creating/deleting the required OpenEBS Volume.\n2. A node component that runs as a DaemonSet, implementing the CSI node services. The node component is responsible for performing the iSCSI connection management and connecting to the OpenEBS Volume.\n\nThe node components make use of the host iSCSI binaries for iSCSI connection management. Depending on the OS, the spec will have to be modified to load the required iSCSI files into the node pods.\n\nVerify that the OpenEBS CSI Components are installed, one node service pod comes up for each node:\n\n $ kubectl get pods -n openebs -l role=openebs-jiva-csi\n\n NAME \t READY STATUS\tRESTARTS AGE\n openebs-csi-controller-0 6/6 \t Running 0 \t 6m14s\n openebs-csi-node-56t5g 2/2 \t Running 0 \t 6m13s\n\n## Before you provision a volume:\n\n**1. (Optional) Create JivaVolumePolicy (jvp)**\n\nCreate Jiva volume policy to configure various tunables required in case of day 2 operations, for example, replicationFactor (defaults to 3), resource limits, tolerations, node-selector, etc.\n\nAlthough It's not mandatory to create JivaVolumePolicy until the user wants to tune some specific default volume policies. If not provided, jiva operator uses the default policies.\n\n apiVersion: openebs.io/v1alpha1\n kind: JivaVolumePolicy\n metadata:\n name: example-volume-policy\n namespace: openebs\n spec:\n replicaSC: openebs-hostpath\n enableBufio: false\n autoScaling: false\n target:\n replicationFactor: 1\n\n $ kubectl get jvp -n openebs\n NAME AGE\n example-volume-policy 1m\n\n**2. Create Storage Class**\n\nCreate Storage to dynamically provision volumes using Jiva CSI driver with volume policy created above.\n\nRead more about various [jiva volume policies](https://github.com/openebs/jiva-operator/blob/master/docs/tutorials/policies.md).\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-jiva-csi-sc\n provisioner: jiva.csi.openebs.io\n allowVolumeExpansion: true\n parameters:\n cas-type: “jiva” \n policy: \"jiva-policy\" // optional\n\n## Volume Provisioning with FIO App Deployment\n\nRun your application by specifying the above Storage Class for the PVCs. For example, we will use the FIO application for demonstration purpose:\n\n**1. Create PVC:**\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: jiva-csi-demo\n spec:\n storageClassName: openebs-jiva-csi-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n\n**2. Deploy FIO Application**\n\nWhen using Pods with PersistentVolumes, we recommend that you use a workload controller (such as a Deployment or StatefulSet). While you would not typically use a standalone Pod, the following example uses one example deployment for simplicity.\n\nThe following example consumes the volume that you created in the previous section:\n\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: fio\n spec:\n selector:\n matchLabels:\n name: fio\n replicas: 1\n strategy:\n type: Recreate\n rollingUpdate: null\n template:\n metadata:\n labels:\n name: fio\n spec:\n containers:\n - name: perfrunner\n image: openebs/tests-fio\n command: [\"/bin/bash\"]\n args: [\"-c\", \"while true ;do sleep 50; done\"]\n volumeMounts:\n - mountPath: /datadir\n name: fio-vol\n volumes:\n - name: fio-vol\n persistentVolumeClaim:\n claimName: jiva-csi-demo\n\nWhile the asynchronous handling of the Volume provisioning is in progress, the application pod may throw some errors propagated from CAS pods to the application pods:\n\n* Waiting for JivaVolume to be Ready: Implies volume components are still being created\n\n* Volume is not ready: Replicas yet to connect to controller: Implies volume components are already created but yet to interact with each other.\n\nVerify that the FIO application pod is in running state:\n\n $ kubectl get pods\n\n NAME READY STATUS RESTARTS AGE\n Fio 1/1 Running 0 97s\n\n## Volume Expansion\n\n**Notes:**\n\n* Only dynamically provisioned volumes can be resized.\n* Volumes containing a filesystem of `XFS`, `Ext3`, `Ext4` type, or even raw block volume can be resized.\n\n**Steps:**\n\n1. Update the increased PVC size in the pvc spec section\n\n kind: PersistentVolumeClaim\n metadata:\n name: jiva-csi-demo\n spec:\n resources:\n requests:\n storage: 10Gi\n volumeName: pvc-136b015f-dac3-11e9-8980-42010a80006c\n status:\n accessModes:\n - ReadWriteOnce\n capacity:\n storage: 5Gi\n phase: Bound\n\n2. Wait for the updated capacity to reflect in PVC status\n\n kind: PersistentVolumeClaim\n metadata:\n name: jiva-csi-demo\n spec:\n resources:\n requests:\n storage: 10Gi\n volumeName: pvc-136b015f-dac3-11e9-8980-42010a80006c\n status:\n accessModes:\n - ReadWriteOnce\n capacity:\n storage: 10Gi\n phase: Bound\n\n3. Verify the updated size of the mounted volume inside the application pod\n\n```\n# df -h /storage\nFilesystem \tSize Used Available Use% Mounted on\n/dev/sdb \t9.7G \t11.3M 9.7G 0% /storage\n```\n\n## Volume Scaleup\n\n**Notes:**\n\n* Only 1 replica at a time can be scaled up.\n* Scale down is not supported.\n\n**Steps:**\n\n1. Increase the desiredReplicationFactor in the jivaVolume spec section by 1. This will spawn another replica for the sts and the data will be rebuilt on the new replica.\n\n```\napiVersion: openebs.io/v1alpha1\nkind: JivaVolume\nmetadata:\nannotations:\n openebs.io/volume-policy: example-jivavolumepolicy\ncreationTimestamp: \"2021-03-22T12:24:23Z\"\ngeneration: 23\nlabels:\n nodeID: k8s-worker-1\n openebs.io/component: jiva-volume\n openebs.io/persistent-volume: pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nname: pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nnamespace: openebs\nresourceVersion: \"170894664\"\nselfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/jivavolumes/pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nuid: ed2ce0b4-acab-48e8-b659-8753be96f345\nspec:\naccessType: mount\ncapacity: 4Gi\ndesiredReplicationFactor: 3\n......\n```\n\n\n\n","notHasFeatureImage":true,"slug":"provisioning-openebs-jiva-volumes-via-csi"},{"id":8,"title":"Deploying YugabyteDB on Google Kubernetes Engine with OpenEBS","author":"OpenEBS","author_info":"OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes","date":"05-04-2021","tags":["OpenEBS","Open Source","Yugabyte","Cloud Native Gke"],"excerpt":"In this blog post, we’ll walk you through the necessary steps to get a 3 node YugabyteDB cluster running on top of GKE, backed by OpenEBS.","content":"\n[OpenEBS](https://www.openebs.io/) is a CNCF project backed by [MayaData](https://mayadata.io/) that provides cloud-native, open source container attached storage (CAS). OpenEBS delivers persistent block storage and other capabilities such as integrated back-up, management of local and cloud disks, and more. For enterprise cloud-native applications, OpenEBS provides storage functionality that is idiomatic with cloud-native development environments, with granular storage policies and isolation that enable cloud developers and architects to optimize storage for specific workloads.\n\nBecause [YugabyteDB](https://www.yugabyte.com/) is a cloud-native, distributed SQL database that runs in Kubernetes environments, it can interoperate with OpenEBS and many other CNCF projects.\n\n**Wait, what is YugabyteDB?** _It is an open source, and high-performance distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner. Yugabyte’s YSQL API is PostgreSQL wire compatible._\n\nIn this blog post we’ll walk you through the necessary steps to get a 3 node YugabyteDB cluster running on top of GKE, backed by OpenEBS.\n\n[![Deploying YugabyteDB on Google Kubernetes Engine with OpenEBS](/images/blog/deploying-yugabytedb-on-google-kubernetes-engine-with-openebs-video-preview.png)](https://player.vimeo.com/video/530995643?app_id=122963)\n\n\n**Why OpenEBS and YugabyteDB?** \nBecause YugabyteDB is a transactional database often used as a system of record, it needs to be deployed as a StatefulSet on Kubernetes and requires persistent storage. OpenEBS can be used for backing YugabyteDB local disks, allowing the provisioning of large-scale persistent volumes. \n\nHere are a few of the advantages of using OpenEBS in conjunction with a YugabyteDB database cluster:\n\n- There’s no need to manage the local disks as OpenEBS manages them.\n- OpenEBS and YugabyteDB can provision large size persistent volumes.\n- With OpenEBS persistent volumes, capacity can be thin provisioned, and disks can be added to OpenEBS on the fly without disruption of service. When this capability is combined with YugabyteDB, which already supports multi-TB data density per node, this can prove to be massive cost savings on storage.\n- Both OpenEBS and YugabyteDB support multi-cloud deployments helping organizations avoid cloud lock-in.\n- Both OpenEBS and YugabyteDB integrate with another CNCF project, [Prometheus](https://prometheus.io/). This makes it easy to [monitor both storage and the database](/docs/introduction/features#prometheus-metrics-for-workload-tuning) from a single system.\n\nAdditionally, OpenEBS can do [synchronous replication](/docs/introduction/features#synchronous-replication) inside a geographic region. In a scenario where YugabyteDB is deployed across regions, and a node in any one region fails, YugaByteDB would have to rebuild this node with data from another region. This would incur cross-region traffic, which is more expensive and lower in performance. But, with OpenEBS, this rebuilding of a node can be done seamlessly because OpenEBS is replicating locally inside the region. This means YugabyteDB does not end up having to copy data from another region, which ends up being less expensive and higher in performance. In this deployment setup, only if the entire region failed, YugabyteDB would need to do a cross-region node rebuild. Additional detailed descriptions of OpenEBS enabled use cases can be found [here](/docs/introduction/usecases).\n\nOk, let’s get started!\n\n**Prerequisites** \n\n![Yugabyte work flow](/images/blog/yugabyte-work-flow.png)\n\nUsing the latest and greatest versions of the available software (as of this blog’s writing), below is the environment which we’ll use to run a YugabyteDB cluster on top of a Google Kubernetes Engine (GKE) cluster backed by OpenEBS\n\n1. YugabyteDB - [Version 2.5.3.1](https://docs.yugabyte.com/latest/quick-start/install/)\n2. OpenEBS - [Version 2.7.0](https://github.com/openebs/openebs)\n3. A [Google Cloud Platform](https://cloud.google.com/gcp/) account\n\n**Step 1: Setting Up a Cluster on GKE** \nTo deploy YugabyteDB on the Google Cloud Platform (GCP), we first have to set up a cluster using Ubuntu as our base node image.\n\n**Note**: _GKE’s Container-Optimized OS does not come with an iSCSI client pre-installed and does not allow the installation of an iSCSI client. Therefore, we’ll be using the Ubuntu with Docker image type for our nodes._\n\nFor the purposes of this demo, I used the Google Cloud Console to configure my Kubernetes cluster. Aside from the typical defaults, here’s the options under the* Node Pools > default-pool > Nodes* I selected\n\n- **Image Type:** Ubuntu with Docker\n- **Series:** N1\n- **Machine Type:** n1-standard-4 (4 vCPU, 15 GB memory)\n\n![Yugabyte nodes](/images/blog/yugabyte-nodes.png)\n\nClick _Create_ and wait for the Kubernetes cluster to come online.\n\n**Step 2: Configure iSCSI** \nThe iSCSI client is a prerequisite for provisioning cStor and Jiva volumes. However, it is recommended that the iSCSI client is setup and *iscsid* service is running on worker nodes before proceeding with the OpenEBS installation. In order to set up iSCSI, we’ll first need to determine the names of the nodes in our cluster\n\n $ kubectl get nodes\n\n NAME \tSTATUS ROLES \tAGE \tVERSION\n gke-cluster-1-default-pool-be95f6dd-5x65 \tReady \t18h \tv1.18.15-gke.1501\n gke-cluster-1-default-pool-be95f6dd-rs6c \tReady \t18h \tv1.18.15-gke.1501\n gke-cluster-1-default-pool-be95f6dd-t4cp \tReady \t18h \tv1.18.15-gke.1501\n\n Now that we have the names of our nodes, we’ll want to log into each node and enable the iSCSI service.\n\n $ gcloud compute ssh \n $ sudo systemctl enable iscsid && sudo systemctl start iscsid\n\n You can check the status of the iSCSI service using the following command:\n\n $ systemctl status iscsid\n\n iscsid.service - iSCSI initiator daemon (iscsid)\n Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)\n Active: active (running) since Fri 2021-03-26 02:25:42 UTC; 18h ago\n Docs: man:iscsid(8)\n Process: 10052 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)\n Process: 10038 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)\n Main PID: 10059 (iscsid)\n Tasks: 2 (limit: 4915)\n CGroup: /system.slice/iscsid.service\n ├─10057 /sbin/iscsid\n └─10059 /sbin/iscsid\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 systemd[1]: Starting iSCSI initiator daemon (iscsid)...\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 iscsid[10052]: iSCSI logger with pid=10057 started!\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 systemd[1]: Started iSCSI initiator daemon (iscsid).\n\n**Step 3: Install OpenEBS** \nNext, let’s install OpenEBS. I’ve found that the OpenEBS Operator is one of the simplest ways to get the software up and running.\n\n $ kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml\n\nOnce the installation is completed, check and verify the status of the pods. You should something similar to this:\n\n $ kubectl get pods -n openebs\n\n NAME READY STATUS\n maya-apiserver-dd655ff87-rbgmd 1/1 Running\n openebs-admission-server-5965c94767-4h8rc 1/1 Running\n openebs-localpv-provisioner-5495669c66-z46lr 1/1 Running\n openebs-ndm-dss64 1/1 Running\n openebs-ndm-gnv75 1/1 Running\n openebs-ndm-operator-68949644b9-mqvlx 1/1 Running\n openebs-ndm-r5pws 1/1 Running\n openebs-provisioner-544cb85449-w9spl 1/1 Running\n openebs-snapshot-operator-6d65b778dd-79zcn 2/2 Running\n\n**Step 4: Create and Attach Disks to Nodes** \nOur worker nodes need to have disks attached. These disks need to be unmounted and not have a filesystem on them. To accomplish this we’ll need to execute the following commands on each node.\n\n $ gcloud compute disks create disk1 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-5x65 --disk disk1\n\n $ gcloud compute disks create disk2 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-rs6c --disk disk2\n\n $ gcloud compute disks create disk3 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-t4cp --disk disk3\n\n Next let’s verify that our block devices are indeed attached.\n\n $ kubectl get blockdevice -n openebs\n\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-03... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n blockdevice-85... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n blockdevice-b0... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n\n**Step 5: Create a Storage Pool Claim** \nNow that we have the names of our block devices and have verified that they are active, the next step is to create a Storage Pool Claim. We’ll use this to then create a Storage Class, and finally use that for our Persistent Volume Claims. The first step in this chain of steps is to configure our Storage Pool Claim YAML file. In this demo, I’ve named it “cstor-pool1-config.yaml”.\n\n $ vim cstor-pool1-config.yaml\n\n #Use the following YAMLs to create a cStor Storage Pool.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-disk-pool\n type: disk\n poolSpec:\n poolType: striped\n blockDevices:\n blockDeviceList:\n - blockdevice-03e93d010db5169322eb16f3e18e33ed\n - blockdevice-22591882979084d0fe580fe229e0d84f\n - blockdevice-4d1b4bacbeec1650b337c2cfda7e3a48\n ---\n\n Once you’ve figured out how to exit vim, the next step is to create the resource.\n $ kubectl create -f cstor-pool1-config.yaml\n\nWe can verify our storage pool with the following command:\n\n $ kubectl get csp\n\n NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE\n cstor-disk-pool-6cmf 1.85M 9.94G 9.94G Healthy false striped\n cstor-disk-pool-jql6 40.6M 9.90G 9.94G Healthy false striped\n cstor-disk-pool-vbz5 68.2M 9.87G 9.94G Healthy false striped\n\n**Step 6: Create a Storage Class** \nNow that we have a storage pool, let’s configure the YAML file for our storage class. In this demo, I’ve named it “openebs-sc-rep1.yaml”.\n\n $ vim openebs-sc-rep1.yaml\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-sc-rep1\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk-pool\"\n - name: ReplicaCount\n value: \"1\"\n provisioner: openebs.io/provisioner-iscsi\n\nAssuming you have remembered how to exit vim from the previous step, we now need to create the storage class.\n\n $ kubectl create -f openebs-sc-rep1.yaml\n\nFinally, let’s verify the storage class.\n\n $ kubectl get sc\n\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE\n openebs-device openebs.io/local Delete WaitForFirstConsumer\n openebs-hostpath openebs.io/local Delete WaitForFirstConsumer\n openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate\n openebs-sc-rep1 openebs.io/provisioner-iscsi Delete Immediate\n openebs-snapshot... volumesnapshot.external... Delete Immediate\n premium-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer\n standard (default) kubernetes.io/gce-pd Delete Immediate\n standard-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer\n\nAt this point, we are now set up for Persistent Volume Claims.\n\n**Step 7: Install YugabyteDB** \nIn this final step we’ll install a 3 node YugabyteDB cluster running on top of GKE that will be backed by the OpenEBS deployment we just completed.\n\nThe first step is to create a namespace.\n\n_$ kubectl create namespace yb-demo_\n\nNext, let’s install the cluster using Helm.\n\n $ helm install yb-demo yugabytedb/yugabyte --set resource.master.requests.cpu=1,resource.master.requests.memory=1Gi,\\\n resource.tserver.requests.cpu=1,resource.tserver.requests.memory=1Gi,\\\n enableLoadBalancer=True --namespace yb-demo --set storage.master.storageClass=openebs-sc-rep1,storage.tserver.storageClass=openebs-sc-rep1 --set persistence.storageClass=openebs-cstor-disk --wait\n\nNote that in the command above we are specifying the following so that YugabyteDB makes explicit use of OpenEBS:\n\n- _storage.master.storageClass=openebs-sc-rep1_\n- _storage.tserver.storageClass=openebs-sc-rep1_\n- _persistence.storageClass=openebs-cstor-disk_\n\nOnce the installation is complete you should be able log into the PostgreSQL compatible YSQL shell on port 5433 with the following command:\n\n $ kubectl --namespace yb-demo exec -it yb-tserver-0 -- sh -c \"cd /home/yugabyte && ysqlsh -h yb-tserver-0\"\n\n ysqlsh (11.2-YB-2.5.3.1-b0)\n Type \"help\" for help.\n yugabyte=#\n\nYou can also access the basic YugabyteDB web admin portal at:\n\n_http://:7000_\n\n![Yugabyte master](/images/blog/yugabyte-master.png)\n\n**Viewing Services and Ingress** \nA quick and visual way to check out all the services and ingress is to go to the “Services and Ingress” view in the Google Cloud Console. If you’ve made it this far you should see something like this:\n\n![Yugabyte ingress](/images/blog/yugabyte-ingress.png)\n\nNote: I have omitted the “Endpoints” column from the screenshot above, but in your view you’ll be able to see the IPs and ports of the various endpoints.\n\nThat’s it! You now have a 3 node YugabyteDB cluster running on GKE with OpenEBS storage.\n\n**Next Steps** \nAs mentioned, MayData is the chief sponsor of the OpenEBS project. It offers an enterprise-grade OpenEBS platform that makes it easier to run stateful applications on Kubernetes by helping get your workloads provisioned, backed-up, monitored, logged, managed, tested, and even migrated across clusters and clouds. You can learn more about MayaData [here.](https://mayadata.io/)\n\n- Learn more about OpenEBS by visiting the [GitHub](https://github.com/openebs/openebs) and [official Docs](/docs) pages.\n- Learn more about YugabyteDB by visiting the [GitHub](https://github.com/yugabyte/yugabyte-db) and [official Docs](https://docs.yugabyte.com/) pages.\n\n**About the author:**\n\n![Jimmy Guerrero](/images/blog/authors/jimmy-guerrero.png)\n\nJimmy Guerrero, VP Marketing, and Community at YugaByte.","notHasFeatureImage":false,"slug":"deploying-yugabytedb-on-google-kubernetes-engine-with-openebs"},{"id":9,"title":"Repeatable OpenEBS Mayastor deployments and benchmarks","author":"OpenEBS","author_info":"OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes","date":"22-03-2021","tags":["Mayastor","OpenEBS"],"excerpt":"Learn about Repeatable OpenEBS Mayastor deployments and benchmarks","content":"\n## Introduction\n\nOpenEBS is one of the most popular Storage-related projects in CNCF, and the newest addition to OpenEBS - Mayastor, is a missing piece that has been absent from the Kubernetes stack for a long time - Kubernetes-native, high performance, distributed Software-Defined Storage or what is increasingly called Container Attached Storage (CAS).\n\nAs the lead developers of OpenEBS Mayastor, we want to be sure our message of an extremely high performing CAS is not only exciting, but also honest and easy to check. We want every interested user to be able to quickly and easily bring OpenEBS Mayastor up, properly tuned and ready for testing with whatever workload the user prefers to try.\n\nIn order to deliver on that promise, we have started a [“Demo Playground” project, open sourced on Github](https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground). Contributions and feedback are welcome.\n\n\n## OpenEBS\n\nOpenEBS is a project with multiple storage engines, with each engine providing the user with different feature sets as well as different usage and performance characteristics. The currently available options can roughly be split into two categories:\n\n* LocalPV: Excellent for workloads that deal with storage resilience at the application level, creating and managing their own replicas and capable of sustaining the loss of a single or multiple nodes, such as Cassandra, and requiring very good storage performance, especially latency-wise.\n* Replicated storage (cStor, Jiva) - for workloads that are less performance-sensitive and some of the more advanced storage features such as synchronous data replication, snapshots, clones, thin provisioning of data, high resiliency of data, data consistency, and on-demand increase of capacity or performance.\n\nAdvanced features come at the cost of higher latency and lower performance, and yet, technology keeps advancing and trying to get the best of both worlds.\n\n\n## OpenEBS Mayastor\n\nOpenEBS Mayastor delivers on the promise of exciting new technology, utilizing NVMe (not just the disks, but the protocol and standards), NVMEoF, SPDK and io_uring. NVMes inside our servers deliver amazing speeds and latencies, huge numbers of IOPS, and using old SCSI or FC protocols only waste resources introducing overheads. Harnessing SPDK and NVMEoF OpenEBS Mayastor achieves speeds that are close to in-host NVMes, without compromising on workload mobility, resilience, flexibility, and enterprise features.\n\nStill, all this exciting tech needs some proper care before it behaves as it should, and we still have a ways to go before it autotunes and autoconfigures itself just right with the help of Kubernetes and workload operators; and yet, as a user willing to take Mayastor for a spin, there should be no reason to wait, if the tuning and preparation can be automated now.\n\n\n## Introducing: the Automation Playground\n\nThe Automation Playground provides an easy onramp for trying out OpenEBS Mayastor in a cloud or self-hosted environment and attempts to keep the installation process correct, standardized, and consistently reproducible, yet both simple and flexible.\n\nThe Playground utilizes popular and familiar software in order to apply the desired state configuration, as well as following a familiar Jenkins-pipeline-like approach.\n\nThe entire process is split into stages, with each stage extensible, replaceable and skippable, if need be, and each stage is called from a simple bash script, where each step is a function, easily copied into a CI engine as a pipeline stage.\n\nThe user experience is as simple as editing a single variables file in order to define the benchmark setup variables and running up.sh. The script will then iterate over the predefined stages, relying on the outcomes of each stage to run the next one\n\nVariables are used to define such things as the setup name (prefixed in all the provisioned resources), user access credentials, Kubernetes installation types, provisioning details, and of course, OpenEBS Mayastor tuning as well as the benchmark itself. For more details, please see the vars file at [https://github.com/mayadata-io/deployment-automation-playground/blob/main/demo-playground/vars](https://github.com/mayadata-io/deployment-automation-playground/blob/main/demo-playground/vars)\n\n\n## Stages\n\nEach software lifecycle consists of several stages - provisioning, deployment, operations, and teardown.\n\nSince we are flexible here, each stage can be skipped if it isn’t required in a given setup.\n\nWhen running a benchmark on a set of self-hosted bare metal machines, the provisioning stage is not needed.\n\nIf Kubernetes is already installed, the Kubernetes installation stage can be skipped.\n\nWhen running the Demo Playground on a host that has direct access to the machines executing the benchmark, the VPN stage can be skipped.\n\nThe only truly essential stages are node preparation and the actual OpenEBS Mayastor workload playbooks that will be installed.\n\n\n#### Stage 1: Provisioning\n\nAt this step, we use Terraform to create a separate environment for the benchmark. Currently, the supported provisioning options are Azure and AWS EC2, with GCP support not too far behind. As a reminder, contributions (and feedback) are welcome.\n\nTerraform is used to create a separate VPC (in EC2) or Resource Group (in Azure), where networking is configured, and VMs are provisioned as per the definitions in the vars file.\n\nThe nodes provisioned are of three varieties\n\n* Master nodes (for Kubernetes Masters)\n* Worker nodes (Kubernetes workers that will be running the workload - make sure these are powerful enough and include fast networking if you want to be able to stress Mayastor)\n* Storage nodes (Kubernetes workers that will be running Mayastor). These instances should have fast local NVMe disks, which means LXs_v2 on Azure, m5d/m5ad/m5dn/i3 on AWS or n1/n2_standard with added Local-SSDs on GCP.\n\nWhen provisioning is complete, an Ansible inventory file is generated by Terraform, to be used in later stages. The inventory contains all the node IPs split into groups and adjusted for the various Kubernetes installers in use.\n\nIf the provisioning stage is skipped, the user must provide the inventory.ini file in the workspace directory, with the file containing the [mayastor_clients] (non-storage workers) and [mayastor_storage] (storage nodes) groups.\n\n#### Stage 2: Start VPN\n\nThis is a small stage, only required when the host executing Demo Playground is not inside the same subnet as the cluster nodes. The stage starts sshuttle after creating a script in the workspace directory. Sshuttle is described as a “poor man’s VPN” - an easy to use package that will tunnel all traffic for a given subnet through an SSH tunnel to a Bastion host.\n\nDuring provisioning, the first Kubernetes Master host has designated the Bastion and will be used for this purpose, effectively working as a VPN concentrator for the Demo Playground setup, placing the executor host in the same subnet as the Kubernetes nodes.\n\n#### Stage 3: Kubernetes setup\n\nAt this step, the Playground will deploy a pre-configured version of Kubernetes on the hosts as described in the inventory. If Provisioning was skipped, this means that the inventory file will have to be expanded with groups that are pertinent to the Kubernetes deployment in use; otherwise, the inventory generated in the Provisioning stage will contain all the required groups.\n\nCurrently two installation types are supported with more planned:\n\n* Kubespray - a well known Ansible based feature rich Kubernetes installer\n* K3S - a simplified and downsized Kubernetes distribution, which can be perfect for a small demo setup. This is also installed via Ansible.\n\nAt the end of the step, the script will extract the KUBECONFIG credentials file from a Master node and place it under workspace/admin.conf. If this stage is skipped, the user will have to extract and add this file manually.\n\n#### Stage 4: Node preparation\n\nIn order to run OpenEBS Mayastor as well as other OpenEBS storage engines, some prerequisites need to be applied to the Kubernetes workers, both the storage and client nodes.\n\nThis includes making sure the iSCSI and NVMeo-TCP client packages are present, installing and enabling the various Linux kernel modules, enabling hugepages, and so on. Some of these settings might require a host restart.\n\nThe stage is implemented as an Ansible playbook, which allows it to reach into the hosts directly in order to prepare them, performing some actions a Kubernetes pod has limited access to.\n\nAt this point, we should have a working Kubernetes setup, with the different worker nodes prepared for using Mayastor either as storage hosts or storage clients.\n\n## Playbooks\n\nActually, the proper stages end at Node Preparation, and then the playbooks take over. The vars file contains a PLAYBOOKS variable, which lists all the playbooks the Playground will apply in sequence.\n\nCurrently, there is one playbook relevant to testing Mayastor - mayastor.yml\n\nBut the script will attempt to run any playbooks mentioned from the deployments directory one after another.\n\nThe Mayastor playbook follows the Mayastor installation instructions, creating the Kubernetes manifests and applying them to the setup, so that all the relevant Mayastor pods, DaemonSets, StorageClasses, Pools etc. are created in the Mayastor namespace, PVCs are created and ready to be used by the user’s workload.\n\nThe Mayastor playbook also contains an optional FIO test, which will create an FIO pod using the first created PVC and run a quick 1-minute benchmark.\n\n## Conclusion\n\nThe Demo Playground project is still in very early stages, and we invite everyone to use, contribute and expand upon it. The goal here is to give the user interested in giving OpenEBS Mayastor a try, a ready tool that does the job in an open, honest, consistent, and reproducible manner.\n\nThe project’s flexibility allows for anyone to add in additional playbooks, which will deploy and run different workloads on top of Mayastor, and we intend to expand upon it, adding some workloads of our own beyond the basic FIO benchmark.\n\nPlease visit us at [https://mayadata.io](https://mayadata.io) and give the Demo Playground a spin at [https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground](https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground).\n\nYou can also find my colleagues and me spending time on the Kubernetes #OpenEBS slack, or at a [Discord room](https://discord.com/invite/zsFfszM8J2) set up to focus mostly on open source collaboration with Mayastor developers (Rusticians may be especially interested), and on the Data on Kubernetes community where a huge variety of users of Kubernetes for data are sharing their perspectives (https://dok.community/).\n\n\n****About the author:****\n\n![Dan Yasny](/images/blog/authors/dan-yasny.png)\n\nDan Yasny is a Principal Field Engineer at MayaData, previously he worked as a Field Engineer at ScyllaDB, an SDET, Technical Product Manager and a Sustaining Engineer at Red Hat, working on such projects as ScyllaDB, Kubernetes, OpenShift, KubeVirt, OpenStack, oVirt/RHV and more.","notHasFeatureImage":false,"slug":"repeatable-openebs-mayastor-deployments-and-benchmarks"},{"id":10,"title":"How are TikTok, Flipkart, KubeSphere, and others using OpenEBS for Local Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"12-03-2021","tags":["LocalPV","OpenEBS","Flipkart","TikTok","Kubernetes","Mayastor","MayaData"],"excerpt":"How are TikTok, Flipkart, KubeSphere, and others using OpenEBS for Local Volumes","content":"\n**How to select the right local volume for your workloads?**\n\nWe have recently seen a massive increase in the usage of different flavors of OpenEBS Local PV. We estimate by looking at container pulls for underlying components combined with some call home data for those users of OpenEBS that enable the capturing of metrics that the weekly new deployments of OpenEBS for LocalPV increased by nearly 10x during 2020. This can be attributed to the fact that more and more cloud native Stateful applications are moving into Kubernetes\n\n![LocalPv Deployment](/images/blog/local-pv-deployment.png)\n\nSome of the prominent users of OpenEBS Local PV include the CNCF, Optoro, ByteDance / TikTok, Flipkart, and many more. You can always read more about OpenEBS users on the OpenEBS.io website and on the GitHub project page here: [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md).\n\nWhile Kubernetes provides native support or interfaces for consuming Local Volumes, the adoption of OpenEBS for LocalPV management suggests that some capabilities are missing that are desired by users. At a high level, dynamic provisioning and the simplicity of deleting Local Volumes are two reasons often given for the preference of some users for the use of OpenEBS LocalPV.\n\nIn this blog, I outline the various types of Local Storage that users have in their Kubernetes clusters and introduce the various flavors of OpenEBS Local PV being used.\n\nBefore getting into the flavors of OpenEBS Local PV, it might be worthwhile to know what Kubernetes offers or means by a Local Volume.\n\n_A [Kubernetes Local Volume](https://kubernetes.io/docs/concepts/storage/volumes/#local) implies that storage is available only from a single node. A local volume represents a mounted local storage device such as a disk, partition, or directory._\n\nSo, it stands to reason - as the Local Volume is accessible only from a single node, local volumes are subject to the availability of the underlying node. If the node becomes unhealthy, then the local volume will also become inaccessible, and a Pod using it will not be able to run.\n\nHence, Stateful Applications using local volumes must be able to tolerate this reduced availability, as well as potential data loss, depending on the durability characteristics of the underlying disk.\n\nAs it happens, many of the Cloud Native Workloads - are distributed in nature and are typically deployed as StatefulSets with multiple replicas. These can sustain the failure or reduced availability of a single replica. MinIO, Redis, PostgreSQL, Kafka, Cassandra, Elastic are just some examples that are deployed using Local Volumes. For these applications - performance and consistent low latency, and ease of management are more important than the resiliency of a node to failures.\n\nAs the large SaaS provider, [Optoro](https://github.com/openebs/openebs/blob/master/adopters/optoro/README.md) puts it: \n*The vast majority of applications are able to better handle failover and replication than a block level device. Instead of introducing another distributed system into an already complex environment, OpenEBS's localPVs allow us to leverage fast local storage. … OpenEBS has allowed us to not introduce a complicated distributed system into our platform. The adoption has been smooth and completely transparent to our end users.*\n\n## Limitations of Kubernetes LocalPV\n\nKubernetes expects users to make Persistent Volumes (PVs) available that it can then associate with PVCs during scheduling. Kubernetes does not help with dynamically creating these PVs as the applications are launched into the cluster.\n\nThis pre-provisioning can become an issue when companies have more than two people or teams managing the Kubernetes clusters, and the Application teams depend on the Kubernetes cluster administrators for provisioning the Volumes.\n\nWe have seen that cluster administrators are challenged by the following aspects:\n\n(a) The type of storage available on the Kubernetes nodes varies depending on how the Kubernetes nodes are provisioned. Available storage types include:\n\n- Nodes have only OS disks with large space that can be used for provisioning Local Volumes.\n- Nodes have one or two additional devices (SSDs or Disks) attached that can be used for provisioning Local Volumes.\n- Nodes have 8 to 16 high-performing NVMe SSDs.\n\n(b) And then, there is a matter of capacity available from the Local Storage and how to manage this to enable the freedom of developers and other consumers of capacity while retaining a level of oversight and assistance by centralized teams:\n\n(c) First, the platform or other centralized team may not know exactly what the capacity a particular team or workload needs - and the developer or data scientist may not know either. Dynamic provisioning within quotas means that users can keep moving without opening a ticket or having a conversation.\n\n(d) Secondly, there are many common operations tasks that need to be performed. Just because the applications are resilient does not mean these tasks entirely disappear. Administrators still would like to safeguard the data with best practices from years of experience in dealing with data such as:\n\n- Enforcing Capacity Limits/Thresholds\n- Securing the Volumes\n- Carving out the Local Volumes from well known or familiar file systems like LVM, ZFS, XFS, and so forth\n- Encrypting the Volumes\n- Enforce compliance with BCP by taking regular snapshots and full backups\n\nThis is where Kubernetes itself stops, and plugins like OpenEBS LocalPV options step into the auto-magically provision and manage the Local Volumes.\n\n## Selecting your LocalPV\n\nOpenEBS provides different types of Local Volumes that can be used to provide locally mounted storage to Kubernetes stateful workloads. The choice of the OpenEBS Local Volume depends on the type of local storage available on the node and the features required.\n\n- OpenEBS Hostpath Local PV - The use of the host path is the simplest, most used, and lowest overhead solution. This approach creates Local PVs by creating a sub-directory per Persistent Volume. This offers flexibility to create different classes of storage and allows administrators to decide into which parent or mounted directory the Persistent Volumes sub-directories should be placed. For example - a storage class for critical workloads vs. non-critical transient workloads, SSD vs. Hard Disk mounted paths, and so forth.\n- OpenEBS Raw file Local PV - The OpenEBS Raw file approach evolved out of the Hostpath approach due to considerable feedback from some OpenEBS community members. Raw file Local PV offers all the benefits of Hostpath Local PV - and in addition, Hostpath supports enforcing Capacity Quotas on Volume subdirectories by creating sparse files per volume.\n- OpenEBS Device Local PV - Device Local PV is best suited for cases where either a complete device or a partitioned device needs to be dedicated to the pod. Workloads like Cassandra or Kafka that need high throughput and low latency often use dedicated device Local PV.\n- OpenEBS ZFS and LVM Local PV - Both ZFS and LVM are selected by seasoned storage administrators that want to leverage all the good things of well-known filesystems or volume management along with the power of Local Volumes. This category offers features like full/incremental snapshots, encryption, thin-provisioning, resiliency against local disk failures by using software raid/mirror, and so forth. Incidentally, you can easily cause a fairly reasoned argument by asking users and community members, and even our own engineers to share their opinions about whether ZFS or LVM is more useful; I'm very happy that the community has progressed to the point that both solutions are now supported and widely deployed.\n\nI hope this overview of LocalPV options and OpenEBS Local has been useful. I plan to follow this with further blogs that get into the details of each flavor of the OpenEBS Local PV.\n\nIn the meantime, you can get started easily with [OpenEBS Local PV](/docs), and the community is always available on the Kubernetes Slack #openebs channel.\n\nOr read more on what our OpenEBS users and partners have to say about Local PV. From our friends at 2nd Quadrant (now part of EDB): [Local Persistent Volumes and PostgreSQL usage in Kubernetes](https://www.2ndquadrant.com/en/blog/local-persistent-volumes-and-postgresql-usage-in-kubernetes/)\n\nAnd from one of the most broadly deployed Kubernetes distributions, Kubesphere: [OpenEBS Local PV is default Storage Class in Kubesphere](https://github.com/openebs/openebs/tree/master/adopters/kubesphere)\n\nOr, again, you can find more stories and can add your own to Adopters.MD on the OpenEBS GitHub: [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md)\n","notHasFeatureImage":false,"slug":"how-are-tiktok,-flipkart,-kubesphere,-and-others-using-openebs-for-local-volumes"},{"id":11,"title":"OpenEBS NDM, go-to solution for managing Kubernetes Local Storage","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"13-01-2021","tags":["OpenEBS"],"excerpt":"Read about OpenEBS NDM, the go-to solution for managing Kubernetes Local Storage.","content":"\nEver since Local Volumes have become generally available (GA) in Kubernetes 1.14, the use of Local Volumes has skyrocketed. This can be attributed to the nature of cloud-native workloads distributed in nature and can sustain node failures. The bare metal underpinning Kubernetes clusters, both on-prem and cloud, can now be configured with local storage to manage stateful workloads. Kubernetes doesn’t treat storage like a native resource on par with CPU or Memory, making it a little difficult to make Kubernetes work out of the box to create effective node-attached storage. OpenEBS NDM helps alleviate this gap by discovering the different storage types attached to each worker node and then creating Kubernetes resources called block devices.\n\nApplication or storage operators can then use the information exposed via block devices to determine how to orchestrate the workloads best.\n\nOpenEBS NDM (Node Device Manager) has been declared GA after being deployed in production for several months as part of the OpenEBS control plane. With the release of version 1.0, NDM adds out-of-the-box support for partitions, LVMs, LUKS encrypted devices, in addition to the unique identification of virtual disks within the cluster. Now offering support for partitions, a single disk can be partitioned. Each partition will be considered a separate block device used by different storage engines like cStor / local PV. NDM also tracks the movement of the devices within a cluster across the nodes.\n\n## Key Storage Problems solved by NDM\n\n* Local Storage Discovery - detecting partitions, devices used as a LUKS device or LVM device, or if it can be accessed as a raw block device.\n* Cluster-wide storage visibility\n* Detect the movement of storage devices across nodes\n* Book-keeping/storage asset management - allocating/reserving, which type of storage should be provided to which workloads.\n\n## Getting Started with NDM\n\nLet us see how NDM helps detect the block devices in the cluster with 3 nodes, each having a completely different disk configuration. The Disk configuration of the nodes are as follows:\n\nMaster: 2 virtual disks\n\nWorker1: 3 virtual disks, one being used by LUKS and two other disks which are partitioned, several partitions are being used as PV's by the LVM.\n\n![](https://lh3.googleusercontent.com/7r1RKQF4udqvigbryA6XFOxRuoOccQSFqhM5C_e27ArTSXnsXIXZk7b3lwgJm4C2VxxWj4rHoED-pZl4PS_KVkF_SC4D2-NLJzokpg2cqlP2upSNva5PLCaBKtQCBueUhWFYTtS9)\n\n\nWorker 2: 4 physical disks\n\n* Deploy NDM into the Kubernetes cluster along with OpenEBS LocalPV\n ```\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-lite.yaml\n ```\n (The latest helm charts for deploying NDM are available [here](https://openebs.github.io/node-disk-manager/))\n\n* Once deployed, check the blockdevices present in the cluster using\n ```\n kubectl get bd -n openebs -o wide\n ```\n\n![](https://lh4.googleusercontent.com/v-iVUrfW6v3wSaXmb06pbek7as_RfFTlRJCmsPzhmId460JIsP0LvXVDBkA0FUnBdO3yt203HqHIBYorT-nP6ZtCZTKdRcao0Ws3tlNyvz8yQF9ytQN_UXxbyO9ZFs6-PeLYHQOD)\n\nSome block devices show partitions that did not exist initially. E.g., sdb1 instead of sdb. This is because NDM creates a partition on virtual disks to identify the disk uniquely. Also, block device resources are now created for LVMs and LUKS encrypted devices. All the block devices listed above will now be treated as individual devices and can be used by any storage engine.\n\n* Deploy a sample application to use the block device\n\nDownload the minio yaml and apply it. (NOTE: A node selector has been added to the minio application pod so that it gets scheduled on worker-1)\n```\nkubectl apply -f [minio-official.yaml](https://gist.githubusercontent.com/akhilerm/194a1606c514d8930addcaef56f9f19f/raw/7d339e5042b4e5e958dde558f1f3509e26c214f3/minio-official.yaml)\n```\nNow check the status of block devices again \n\n![](https://lh3.googleusercontent.com/A_JL0jXsZhmIPPrRYCSeMHVcPsey6ahFYV1_LVUapmbPLTrcgGEAao_ohbx9zU_SZl-lHmKGYgdMqh4czUCISSezbcOi4rznQNuX0sTAomO4y5HQLVYicTD4s1mPOZfUciacEOU_)\n\nWe can see that the device `dm-2`, is the LUKS device, has been claimed and used by the application.\n\n* Pool movement across nodes\n\n NDM helps in seamlessly moving cStor pools from one node to another. Whenever the devices that constitute a pool are moved from one node to another (disconnecting disks from one node and reconnecting on another), the block device resource is updated with the latest information. NDM tracks this movement. cStor can use this information to migrate pools as required.\n\n* Reserving storage for workloads\n\n NDM provides a feature to reserve devices for certain workloads. E.g., Users can reserve all SSDs for a performance intensive workload. This reservation is achieved using block-device-tags. More information on using block-device-tags with LocalPV can be found [here](/docs/user-guides/localpv-device#optional-block-device-tagging).\n\n## Future Roadmap\n\n* Southbound provisioning\n* Metrics (currently in alpha)\n* API Service to interact with NDM\n* Ability to create partitions or LVM volume groups - preparing storage in general.\n\n## Interested in Contributing to NDM?\n\nNDM is an OpenEBS project, which itself is a CNCF sandbox project. [OpenEBS on GitHub](https://github.com/openebs/node-disk-manager) is a great place to join if you want to contribute to our codebase. You can also interact with us on the OpenEBS channel in [Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F).","notHasFeatureImage":false,"slug":"openebs-ndm,-go-to-solution-for-managing-kubernetes-local-storage"},{"id":12,"title":"Storage is Evolving!","author":"Nick Connolly","author_info":"Nick is the Chief Scientist at MayaData and a pioneer of storage virtualization, holding patents ranging from highly-scalable algorithms through to data protection techniques.","date":"11-12-2020","tags":["OpenEBS"],"excerpt":"Learn how storage has evolved over the years. ","content":"\nBefore the turn of the century, storage systems were typically controlled by dedicated firmware running on custom hardware. These proprietary systems were time-consuming to design, expensive to build, and resistant to innovation.\n\nIn 1998, Software-Defined Storage was pioneered by DataCore Software with its SANsymphony suite of products, based on the realization that general-purpose computers had become fast enough to handle the demands of a high-performance storage stack. For context, this was an era when a system with more than two cores was a rarity and both memory and storage were measured in MBs! The primary protocol in use in the enterprise was SCSI, whether directly connected or accessed through a Fibre Channel network, response times were measured in the tens of milliseconds, and accessing storage over Ethernet using iSCSI was only just starting to be worked on.\n\n## The hardware environment is changing!\n\nIn the last few years, the hardware environment has changed significantly. Instead of the relentless drive for ever-increasing clock speeds, systems with over a hundred cores are now mainstream. Developing highly-performant algorithms that operate at this scale of parallelism is a complex and time-consuming process that, generally speaking, is uneconomic to pursue. Storage media has also undergone a transformation, with SSDs based on flash memory delivering orders of magnitude better performance than spinning disks. Their response time, which can be measured in microseconds, has highlighted the inefficiencies of the decades-old SCSI protocol.\n\nNVMe is a ‘state of the art’ storage protocol for a new era. Designed from the ground up for maximum parallelism and lock-free operation, it offers up to 64k independent I/O queues each with 64k entries and a simplified command set. Connected over PCIe, it delivers low latency and high bandwidth data directly to an application, enabling it to fully utilize the capabilities of the underlying flash memory. NVMe over Fabrics (NVMe-oF) provides network access to remote storage and targets less than 10 microseconds in additional latency.\n\n## Application development is changing!\n\nRather than building the large monolithic codebases that were the norm at the turn of the century, modern development practices are based around composable architectures; containerized microservices that scale dynamically to meet performance requirements. For more background on this trend, see my [earlier post](https://www.datacore.com/blog/5-changes-that-are-reshaping-software-development/) and the excellent articles in [MayaData’s blog](https://blog.mayadata.io/). Kubernetes is rapidly becoming the control plane for the enterprise.\n\n## A New Era\n\n![New Era](https://lh3.googleusercontent.com/5C8pUrteH4V8JB1li4myidOdIP1xAefDES3ksqG1SaxFX4YHhFZz2gX-tNQV7n4UVuHS-BvZejBVnDnLJiwte6LgGgHN2dzsKDKxC2cd-popha9Ljnw9CWNQ2JUvL_1a2F-w8x0i)\n\nA new era requires a new kind of storage stack! A stack that is based around today’s technologies rather than being anchored to the last century. A stack that is portable and flexible. A stack that supports rapid innovation. That delivers the performance that applications require.\n\n## Container Attached Storage\n\nThe new category of [Container Attached Storage](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/), of which OpenEBS is the de-facto open source standard, orchestrates the storage stack with the same flexibility as the application. Implemented as a microservices based architecture, it runs within Kubernetes and gives users the freedom to define the way that they want to access, protect, and manage their data. The days of the dedicated storage administrator are coming to an end!\n\nFor Mayastor, the latest storage engine to be added to OpenEBS, flexibility, and performance are achieved by basing the stack around the [Storage Platform Development Kit (SPDK)](https://spdk.io/), which provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. Based on the NVMe protocol, it delivers blistering performance from today’s hardware as well as being ready for the next generation of Intel Optane based SSDs that are just becoming available. For more details, see some [recent results](https://openebs.io/blog/mayastor-nvme-of-tcp-performance/).\n\n## Microsoft Windows\n\nHowever, amid all the discussions about flexibility and portability, there is one small footnote that often goes unnoticed: ‘not *supported on Windows*’. It’s understandable, because most of the projects that are shaping this new era have their roots on Linux or FreeBSD, but it overlooks the sheer scale of Windows Server deployments in enterprise environments. Things are changing, with significant investments being made in Kubernetes on Windows, but it’s a slow process; one project at a time!\n\nMayaData’s mission is to enable data agility - so we were uncomfortable with our high-performance Container Attached Storage solution, OpenEBS Mayastor, not being available on Windows platforms. With that in mind, we have created the [Windows Platform Development Kit (WPDK)](https://github.com/wpdk/wpdk) to act as a foundational layer to make it easier to port the SPDK to Windows. In addition, we are working with the SPDK community to make a few changes to the code base to support this. It is a testament to the quality of the excellent SPDK project that so few changes have been required so far.\n\nThe project also benefits from the work done by the DPDK on Windows community who has invested a significant amount of time porting the underlying [Data Plane Development Kit (DPDK)](https://www.dpdk.org/), a Linux Foundation project that consists of libraries to accelerate packet processing workloads running on a wide variety of CPU architectures.\n\n## Windows Platform Development Kit\n\n![Windows Platform Development Kit](https://lh4.googleusercontent.com/UDp5t-uCJeM6QlsMpoZCz-oxp2CyYDPS1BMhkdeaXn4asIPhdLzy0GLG74xdceDyWAa8bCrijsMLOZfrwKC7vQyQLNS-uGJbGLXyeDtBljMvMNDQphRtcfgMJ65mhZBTC7v6wFwg)\n\nThe MayaData developed and contributed Windows Platform Development Kit has currently reached ‘alpha’. Most of the required functionality is believed to be present, unit tested, and working correctly, but there are still areas that need further development.\n\nIt is possible to build the SPDK tree, run the associated unit tests, serve an iSCSI target on Windows, and mount it as a volume.\n\nIt is anticipated that this collaboration will deliver the following benefits to Windows users:\n\n1. Enable high-performance access to NVMe storage directly from applications.\n2. Native software-defined storage stacks, including OpenEBS Mayastor.\n3. Support for NVMe-oF adaptors from manufacturers such as Mellanox and Broadcom.\n\nThe Windows Platform Development Kit is open source, under a BSD-3 clause license. Community contributions are welcomed and needed! To get started please head to [https://wpdk.github.io](https://wpdk.github.io) or access the WPDK code and documentation on [GitHub](https://github.com/wpdk/wpdk).","notHasFeatureImage":false,"slug":"storage-is-evolving!"},{"id":13,"title":"OpenEBS on DigitalOcean Marketplace","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"3-12-2020","tags":["OpenEBS"],"excerpt":"Learn how to deploy OpenEBS on the DigitalOcean marketplace","content":"\nDeploying OpenEBS on DigitalOcean can directly be done from the console. DigitalOcean provides the feature to create a cluster with OpenEBS deployed on it already. To get started, follow the below-mentioned steps:\n\nWORKFLOW:\n\n![Workflow](https://lh3.googleusercontent.com/fQOb_mUG5ebZ4eu2eLCZw4WFIiG_LOUgk2xXj4tBsXokE1oMu5H4SQDcx1jgbpLYBBn4gVpeDOwgU_DhagUjyHi4_kFL3evGUjVTIfkY3Xdf6071c6XWO6AoJ5PruG5f1njtvaJm)\n\nSTEP 1: Getting started \nLogin to your [DigitalOcean](https://cloud.digitalocean.com/login) account.\n\nSTEP 2: Creation of cluster \nOnce you log in, you arrive at the dashboard, click on Marketplace under DISCOVER located on the left sidebar.\n\nNext, scroll down to find OpenEBS. On clicking, you will be redirected to a page where you will find the details about OpenEBS and the Create OpenEBS button on the right side.\n\nNext, you need to provide the necessary details such as Data Center region, cluster capacity, cluster name, etc. (It is advisable to provision 3 nodes with 4vCPUs and 8 GB memory to ensure that the resources are sufficient at all times.)\n\n![Creation of cluster](https://lh3.googleusercontent.com/zvcGjrkGegKSp-t29NQf5XPHzCf6LqRn-XFJuRsxNZBfopwNibKiUwBDo9KSFGTWub6tEqLnl_IWKtCTykql315aqUlTqa7U1ORYJ5H4OmIhdVeHArPmRELvKk94GFLIbui9LJTx)\n\nSTEP 3: Connecting your cluster\nCreation, resizing, and deletion can be carried out from UI, but you require command-line tools from your local machine or a remote management server to perform administrative tasks. The detailed steps to install the management tools and connect the cluster to your local machine can be found under the Overview section.\n\n![Connecting your cluster](https://lh4.googleusercontent.com/5ftx1DgzSOKXRQ_UdiMmakqRMdOMnyes7n0l7kT23t50dSloHosbqAgx7zH2hqaMhE77KIoKINERafGDCyPgZbvnGNQ27oIvpeNm7YqCjGv-6lx9aAgQMSoHiE4j8BrdYABdPg7K)\n\nTo verify, execute the following command:\n\n```\n$ kubectl get ns\n```\n\nOutput:\n```\nNAME STATUS AGE\ndefault Active 13m\nopenebs Active 13m\n```\nThe output must contain openebs ns in an Active state.\n\nNext, execute:\n\n```\n$ kubectl get pods -n openebs\n```\n\nOutput:\n```\nNAME READY STATUS RESTARTS AGE\nopenebs-admission-server-5c4d545647-r4vgr 1/1 Running 0 13m\nopenebs-apiserver-56f77c9965-xft68 1/1 Running 2 13m\nopenebs-localpv-provisioner-64c67b5b89-czv8b 1/1 Running 0 13m\nopenebs-ndm-5f6nt 1/1 Running 0 13m\nopenebs-ndm-74njq 1/1 Running 0 13m\nopenebs-ndm-operator-7bc769dcff-b45bc 1/1 Running 1 13m\nopenebs-ndm-spttv 1/1 Running 0 13m\nopenebs-provisioner-755f465f9d-fr67l 1/1 Running 0 13m\nopenebs-snapshot-operator-7988fc8646-zpd98 2/2 Running 0 13m\n```\nAll the pods must be in a running state.\n\nSTEP 4: Attaching BlockDevices \nTo attach BlockDevices to the created nodes, click on Volumes on the left sidebar and then click on the Add Volume button.\n\n![Adding Volume](https://lh3.googleusercontent.com/D96l0ASgsYCKEoenZQS7r-i_bdmLMlQ2PxcxYGqYLilrFospNmLmnVwfZAT-VYBHvSP31U70bgjdo0WhUbSuDfM0mU84s3-BopEd0vxuEHlZg64cnzIwO7LlLPc9RjhL5ResDD0-)\n\nNext, select the volume size ( provision at least 30 GB), select the node(droplet) to which it gets attached and provides a name, then click on the Create Volume button. Repeat these steps for each node (droplet).\n\n![Attaching BlockDevices](https://lh3.googleusercontent.com/i2IkMHV3CmVK8_fgiWMtiXhqlbkWGyCoCXaz4a0hXAR49WEuzXg6s7lbMEZFGr6oXLLFVAsLTfgJTELlrMKTE4mi5aNjDSMKMZn9XCMCGlWLpeUaaC4VsRg2xFPgw0tXLuG2T2uq)\n\nNOTE:\n\n*For cStor, choose Manually Mount and Format under Choose Configuration Options.*\n\n*For Jiva, choose Automatically Format and Mount under Choose Configuration Options.*\n\n*After the BlockDevices get attached for all the nodes, you can see an output similar to the below image.*\n\n![Output after the BlockDevices get attached for all the nodes](https://lh4.googleusercontent.com/i9KOccFfCGjP-Q3E2KpR0YOV3EXDfTin4RbNZbgo9A0PTSYRUj8E989KqnzHYXMigbfE0FZWK1_V0Jg_lAvZKN9iShkxLIyMFGmO9uVEYWhcosL-xUNc-VnrXpYcbu1VDKE-5zOT)\n\nNext, you have to provision OpenEBS volumes. Click [here](/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume) to know more.","notHasFeatureImage":false,"slug":"openebs-on-digitalocean-marketplace"},{"id":14,"title":"Atlassian Jira Deployment on OpenEBS","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"20-11-2020","tags":["OpenEBS"],"excerpt":"Learn how to deploy Atlassian Jira on OpenEBS in this short post.","content":"\n***Jira*** Software is part of a family of products designed to help teams of all types manage work. Originally, **Jira** was designed as a bug and issue tracker. But today, **Jira** has evolved into a powerful work management tool for all kinds of use cases, from requirements and test case management to agile software development.\n\n## Requirements\n\n#### Install OpenEBS\n\nIf OpenEBS is not installed in your K8s cluster, this can be done from [here](/docs/user-guides/installation). If OpenEBS is already installed, go to the next step.\n\n#### Configure cStor Pool\n\nIf cStor Pool is not configured in your OpenEBS cluster, this can be done from [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools). Sample YAML named **openebs-config.yaml** for configuring cStor Pool is provided:\n\n```\n#Use the following YAMLs to create a cStor Storage Pool.\n# and associated storage class.\napiVersion: openebs.io/v1alpha1\nkind: StoragePoolClaim\nmetadata:\n name: cstor-disk\nspec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get blockdevices -n openebs`\n #\n # `Block devices` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n# Replace the following with actual disk CRs from your cluster `kubectl get blockdevices -n openebs`\n# Uncomment the below lines after updating the actual disk names.\n blockDevices:\n blockDeviceList:\n# Replace the following with actual disk CRs from your cluster from `kubectl get blockdevices -n openebs`\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\n---\n```\n\n#### Create Storage Class\n\nYou must configure a StorageClass to provision cStor volume on the cStor pool. In this solution, we are using a StorageClass to consume the cStor Pool, which is created using external disks attached to the Nodes. Since Jira is a deployment application, it requires three replications at the storage level. So cStor volume replicaCount is 3. Sample YAML named **openebs-sc-disk.yaml** to consume cStor pool with cStor volume replica count as 3 is provided:\n\n```\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n name: openebs-cstor-disk\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"3\" \nprovisioner: openebs.io/provisioner-iscsi\nreclaimPolicy: Delete\n```\n\n### Deployment of Jira\n\nSample Jira Yaml:\n\n```\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n labels:\n app: jira\n name: jira\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: jira\n name: jira\n spec:\n containers:\n - name: jira\n image: \"doriftoshoes/jira:7.3.6\"\n resources:\n requests:\n cpu: \"2\"\n memory: \"2G\"\n volumeMounts:\n - name: \"jira-home\"\n mountPath: /opt/jira-home\n volumes:\n - name: jira-home\n persistentVolumeClaim:\n claimName: demo-vol1-claim\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: jira\n name: jira\nspec:\n ports:\n - port: 8080\n targetPort: 8080\n selector:\n app: jira\n type: LoadBalancer\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: demo-vol1-claim\nspec:\n storageClassName: openebs-cstor-disk\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10G\n```\n\nNext, apply both the ***Jira deployment*** and service to your Kubernetes cluster.\n\n```\nkubectl apply -f jira.yaml\n```\n\n#### Verify Jira Pods:\n\n#### Run the following to get the status of Jira pods:\n\n```\nkubectl get pods\n```\n\nFollowing is an example output:\n\n```\nNAME READY STATUS RESTARTS AGE\njira-5xxxxxxxx-2xxxx 1/1 Running 0 1d12h\n```\n\nThat's it for today's blog. Thanks for reading. Please leave your questions or feedback, if any, in the comment section below.","notHasFeatureImage":false,"slug":"atlassian-jira-deployment-on-openebs"},{"id":15,"title":"Mayastor NVMe-oF TCP performance","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"19-11-2020","tags":["Mayastor"],"excerpt":"Overview of using Mayastor to try out some of the fastest NVMe devices currently available on the market and see how we perform on top of these devices within k8s, using the container attached storage approach for which OpenEBS is well known.","content":"\nFor a while now, we have been saying that OpenEBS Mayastor is “high” performance and community members have written [blogs](https://medium.com/volterra-io/kubernetes-storage-performance-comparison-v2-2020-updated-1c0b69f0dcf4) showing that the performance of OpenEBS Mayastor indeed is much better or on par with others even when running on relatively slow cloud volumes. However, is Mayastor high performance or just “as fast” as the other things out there? \n\nIt used to be the case that CPUs were much faster than the storage systems they served. With modern NVMe, this does not ***have*** to be the case anymore. NVMe is a ***protocol*** that can go fast but does not have to be fast. What this means is that you can use NVMe as your transport protocol for any block device, not just flash. Yes, this is what Mayastor does. It is really useful to keep in mind the distinction between NVMe as a protocol and NVMe devices - you don’t need to use them together but, when you do, additional performance can be unlocked.\n\nIn this blog, we will be using Mayastor to try out some of the fastest NVMe devices currently available on the market and see how we perform on top of these devices within k8s, using the container attached storage approach for which OpenEBS is well known. We will show what happens when you marry NVMe as a protocol (embedded within Mayastor) and fast NVMe devices from our friends at Intel.\n\nBefore we get started, you might wonder how we came to this point that a new project like OpenEBS Mayastor was able to deliver amongst the fastest storage available today. Richard Elling of Viking / Sanmina recently wrote an excellent blog on the trends in hardware design that puts NVMe and OpenEBS Mayastor into context:  [https://richardelling.substack.com/p/the-pendulum-swings-hard-towards](https://richardelling.substack.com/p/the-pendulum-swings-hard-towards)\n\n## Hardware setup\n\nLet’s get to it. We will be using three machines that will run kernel version 5.8. The hardware configuration of each host is as follows:\n\n- Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz\n- Intel Corporation NVMe Datacenter SSD [Optane]\n- Mellanox Technologies MT28908 Family [ConnectX-6]\n\n## Baseline performance\n\nTo understand the performance of the device we will be using throughout the test, we run the following Fio workload:\n\n [global]\n ioengine=linuxaio\n thread=1\n group_reporting=1\n direct=1\n norandommap=1\n bs=4k\n numjobs=8\n time_based=1\n ramp_time=0\n runtime=300\n iodepth=64\n \n \n [random-read-QD64]\n filename=/dev/nvme1n1\n rw=randread\n stonewall\n \n \n [random-write-QD64]\n filename=/dev/nvme1n1\n rw=randwrite\n stonewall\n \n \n [random-rw-QD64]\n filename=/dev/nvme1n1\n rw=randrw\n stonewall\n\n![Mayastor NVME](/images/blog/mayastor-nvme1.png)\n\nThese numbers are incredibly high and are provided by a ***single*** device. Note that the benchmark itself is rather synthetic in the sense that, in practice, no workload is 100% random.\n\n## High-level overview of the experiments\n\nMy approach in this benchmarking is very much OpenEBS Mayastor “the hard way”. If you want an easier to use solution that for example automates pool creation and device selection and so on - we call that offering Kubera Propel (also open source btw). You can learn more about Kubera Propel on the [MayaData.io](https://mayadata.io/) website.  \n\nOn two of the nodes, we create a pool (MSP CRD) which we use in the control plane to determine replica placement. To construct pools, we must have what we call a persistence layer. We support several ways to access this persistence layer. To select a particular scheme we use URIs. In this case we will be using today the pcie:/// scheme. This means that Mayastor will directly write into the NVMe devices listed above. The nice thing is that from the user perspective, things do not change that much. We simply replace disks: [‘/dev/nvme0n1’] with disks: [‘pcie:///80:00.0’]. Note that this persistence layer is used to store the replicas of the PVC. Once we have this layer up and running, we will create storage classes and select that we want to use nvmf (NVMe-oF) as the transport layer between the replicas, resulting in NVMe all the way through. \n\nAfter we have deployed mayastor we applied to following two storage classes:\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: nvmf\n parameters:\n repl: '1'\n protocol: 'nvmf'\n provisioner: io.openebs.csi-mayastor\n ---\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: nvmf-mirror\n parameters:\n repl: '2'\n protocol: 'nvmf'\n provisioner: io.openebs.csi-mayastor\n\nNote that `protocol: `nvmf` is just a shorthand for the NVMe-oF format mentioned above. We will be using both storage classes to run a single replica followed by a two way replica AKA mirror.  We use the following YAML to create the volume.\n\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: ms-volume-claim\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 100G\n storageClassName: nvmf\n\nAfter creating the PVC, Mayastor’s control plane creates a CRD, “Mayastor Volume” (MSV), that contains additional information about the corresponding volume.  Using kubectl describe msv -n mayastor we get:\n\n Name: ba081dc3-46db-445b-969c-7e5245aba146\n Namespace: mayastor\n Labels: \n Annotations: \n API Version: openebs.io/v1alpha1\n Kind: MayastorVolume\n Metadata:\n Creation Timestamp: 2020-09-11T08:49:30Z\n Generation: 1\n Managed Fields:\n API Version: openebs.io/v1alpha1\n Fields Type: FieldsV1\n fieldsV1:\n f:spec:\n .:\n f:limitBytes:\n f:preferredNodes:\n f:replicaCount:\n f:requiredBytes:\n f:requiredNodes:\n f:status:\n .:\n f:nexus:\n .:\n f:children:\n f:deviceUri:\n f:state:\n f:node:\n f:reason:\n f:replicas:\n f:size:\n f:state:\n Manager: unknown\n Operation: Update\n Time: 2020-09-11T08:51:18Z\n Resource Version: 56571\n Self Link: /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorvolumes/ba081dc3-46db-445b-969c-7e5245aba146\n UID: 94e11d58-fed9-44c9-9368-95b6f0712ddf\n Spec:\n Limit Bytes: 0\n Preferred Nodes:\n Replica Count: 1\n Required Bytes: 100000000000\n Required Nodes:\n Status:\n Nexus:\n Children:\n State: CHILD_ONLINE\n Uri: bdev:///ba081dc3-46db-445b-969c-7e5245aba146\n Device Uri: nvmf://x.y.z.y:8420/nqn.2019-05.io.openebs:nexus-ba081dc3-46db-445b-969c-7e5245aba146\n State: NEXUS_ONLINE\n Node: atsnode3\n Reason:\n Replicas:\n Node: node3\n Offline: false\n Pool: pool-atsnode3\n Uri: bdev:///ba081dc3-46db-445b-969c-7e5245aba146\n Size: 100000000000\n State: healthy\n Events: \n\n## Results single replica\n\n![Table (% difference of Mayastor overhead)](/images/blog/mayastor-nvme2.png)\n\n![Chart](https://lh5.googleusercontent.com/whpgDl_Id_oo4tUdl7RZDv-C1Uq2ZfvM6Eh7KXbwNkNTu5Mki14meunBgF1PMWMnWLoccSGgHqCfRKXgQpJTfQG42NaS0GkwWRCuNpWGh7znOhqQ94aJXiCODJkzNUs9-G2ucqMJ)\n\nFrom the results we can see that we are very close to the performance of the local device. To be sure we can put it in the right perspective, the difference between the experiments here is that with the baseline, the workload was local. With repl=1 we use the same NVMe device but export it through our pool layer (and thus provide volume management), but also go over the network.\n\n## Results 2 replicas (mirror)\n\nWe are going to repeat the same test, this time, we will use two replicas. As we now have double the disks bandwidth, we expect to see that the read performance will go up. For writes, however, we actually expect a drop in performance, because we must do each write to both disks before we can acknowledge the IO.  Note that Mayastor does not cache - if you read the blog referenced above from Richard Elling you can learn why caching seems to be falling out of favor for use cases in which millions of IOPS are desired.\n\n![Mayastor nvme](/images/blog/mayastor-nvme3.png)\n\n[Chart](https://lh4.googleusercontent.com/GJ7c_cZ6vuDxd9-jAnU3XxAc8L0idA9sscz2JB5XVa0taj8v56H6MSIFB56XqPQzQsy_p49-yHlwhCB8SVjZ05YfT0oRdlFt0EMBze1IDrCioqWgtWypidK9fBpb9p3ULI19Dhfa)\n\n## Wrapup\n\nWith the above experiments, we have demonstrated that with OpenEBS Mayastor we have built a foundational layer that allows us to abstract storage in a way that Kubernetes abstracts compute. While doing so, the user can focus on what's important -- deploying and operating stateful workloads. \n\nIf you’re interested in trying out Mayastor for yourself, instructions for how to setup your own cluster, and run a benchmark like **fio** may be found at [mayastor.gitbook.io/](https://mayastor.gitbook.io/introduction/).\n\nAnd if you are a Kubera Propel user, you’ll find that we’ve productized some of the benchmarking above so that platform teams and other users can programmatically use benchmarks in their decisions about workload placement. We are working with a number of users about operating OpenEBS Mayastor / Kubera Propel at scale. Please get in touch if you have suggestions, feedback, ideas for interesting use cases and so on. Contributions of all kinds are welcome!\n","notHasFeatureImage":false,"slug":"mayastor-nvme-of-tcp-performance"},{"id":16,"title":"Mayastor Engine Reaches Beta","author":"Glenn Bullingham","author_info":"Director of Solution Architecture","date":"19-11-2020","tags":["OpenEBS"],"excerpt":"Mayastor, the storage engine by OpenEBS has reached the beta stage. Read the blog to know more.","content":"\nAs I write this, it is early November, and the winds of change are blowing. The United States has a new president. Here in the United Kingdom, Keats' days of mists and mellow fruitfulness are departing, replaced by a low sun and the first morning frosts. And at MayaData, we see the Mayastor project carefully but tenaciously emerging from alpha/pre-release into its beta phase. In fact, Mayastor now undergirds MayaData’s commercial offering for performance sensitive containerized workloads, called [Kubera Propel](https://mayadata.io/product).\n\n> ***“Beta Software: Software considered to be feature complete and substantially free of known major defects”***\n\nSignificant contributions over the past 18 months have seen the project raised from concept to working software. A major requirement of our MVP specification is that it should carry minimal performance overhead; Mayastor is intended to satisfy demands for performance at all levels of scale. At the beginning of Autumn, working in conjunction with Intel’s labs in the UK, we were able to validate that assertion; deployed in conjunction with the latest generation of Optane NVMe devices, the Mayastor data plane was found to introduce less than 6% overhead; you can read more about that benchmarking [here](https://openebs.io/blog/mayastor-nvme-of-tcp-performance/). Having addressed that performance criteria and the other principle MVP requirements, the Mayastor team at MayaData has begun to focus its contributions to the project on QA as we approach Beta and GA releases.\n\nIn particular, we’ve greatly increased end-to-end test coverage on Kubernetes. How MayaData tests Mayastor is something that I’ll elaborate upon in a forthcoming post.  However, suffice it to say customary suspects feature (Jenkins, mocha, nix, cargo test), whilst we’re also collaborating with our colleagues who maintain the [Litmus Chaos](https://litmuschaos.io/) project. By the time that you’re likely reading this, Mayastor-specific chaos tests should be available to all on [ChaosHub](https://hub.litmuschaos.io/).\n\n## Ease of Use, Perf, and Availability\n\nMayastor MVP requirements center on ease of use, performance, and availability. In the Beta phase and subsequent GA release, these will be realized as a CAS platform with full NVMe data path semantics, declarative configuration via CSI compliant dynamic provisioning, and N+1 synchronous mirroring of data at the persistent layer. This closely approaches functional parity with the current OpenEBS storage engines (cStor, Jiva, and Local PV), with snapshot and cloning functionality to be added in Q1 2021. Mayastor will also very soon be the recipient of a streamlined deployment and configuration experience, which is exclusive to this engine.\n\n## Try it Yourself\n\nIf you’re a member of the Kubernetes community looking to implement a platform-native storage solution in the new year, now is an ideal time to start evaluating Mayastor. The other venerable and respected engines of OpenEBS won’t be retiring overnight, but as full feature parity emerges, MayaData’s commercial products and services will on Mayastor as their default storage engine; we do recognize that some users will continue to prefer various flavors of Dynamic Local PV from OpenEBS - as a recent [blog](https://www.percona.com/blog/2020/10/01/deploying-percona-kubernetes-operators-with-openebs-local-storage/) from the CTO of Percona attests as do countless [Adopter.md case studies](https://github.com/openebs/openebs/blob/master/ADOPTERS.md) including that of the SaaS company Optoro, also a CNCF case study. Mayastor’s roadmap includes provisions for the inward migration of existing OpenEBS users. It’s an equally opportune moment to [consider becoming a contributor](https://github.com/openebs/Mayastor/issues/new/choose) to the project yourself.\n\nTo help with Mayastor onboarding as we prepare to go to full steam, we’re putting together a new documentation site over at[ GitBook](https://mayastor.gitbook.io/introduction/), which includes a comprehensive quick-start deployment guide (developer docs will remain, at least for now, with the OpenEBS/Mayastor GitHub repository). We’re also holding [Office Hours at Kubecon NA 2020 this month](https://kccncna20.sched.com/?searchstring=OpenEBS&iframe=no&w=&sidebar=&bg=), and we’d love to see you there.\n\nIf you’d like to try Mayastor from the source - you can do so, of course, from the GitHub repositories. If you’d like to also try out management utilities, including a cool management interface and available 24x7 support - please take a look at [Kubera Propel](https://go.mayadata.io/register-for-kubera-chaos-and-propel-technical-preview). A free forever for individual use tier is available.\n\n## Conclusion\n\nIt is a propitious time for MayaData and Mayastor - and for data on Kubernetes more broadly. If you have always wanted to run workloads on Kubernetes but were put off by the stories of performance challenges, you can now move forward with confidence. Kubernetes enabled storage with the help of Mayastor performs faster than that of traditional shared everything storage while retaining the ease of use, open source community, and Kubernetes semantics for which OpenEBS has become famous.\n","notHasFeatureImage":false,"slug":"mayastor-engine-reaches-beta"},{"id":17,"title":"Migrate CSPIs to a different node by moving the disks","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"04-11-2020","tags":["OpenEBS"],"excerpt":"Step by step guide to migrate CStorPoolInstances from one node to different nodes by moving the set of underlying disks","content":"\nThis blog describes steps to migrate CStorPoolInstances from one node to different nodes by **moving the set of underlying disks to a new node that participates in the pool provisioning**. There were a couple of use cases where this feature can be helpful:\n\n1. Scaling down and scaling up nodes in the cluster(in a cloud environment) by retaining external volumes(for cost savings).\n2. Replacing failed storage nodes with new nodes by attaching the same old disks to the new node.\n\n**Steps to migrate the CSPI to different node:**\n\n1. Detach the disks belonging to the CSPI that you wish to migrate from the node and attach it to the new node. If you are using a cloud platform, check on their documentation, or ask the administrator about the steps to do this.\n2. Change the node selector in the CSPC YAML (next section describes how to do this).\n\n![](https://lh4.googleusercontent.com/XTwKu6lE3lyoZ3cHRO9HNJGUaTOoGfE-OWGuscrmukbxEKJNPSaEqxUPbbNnnc3dcD-Aybc2_AF0y2Scf0QBxSDG_f9QZWRu67sXZjoMKO6nymhgelEWfDzPjfGKi4D9UwLBaN0D)\n\n**Existing setup**:\n\nI have a three-node cluster with CSPC and CSI volumes already provisioned(To create CSPC pools and CSI volume click [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md#quickstart)). Here is detailed information:\n\n**Cluster details**:\n\n Kubernetes Cluster: AWS\n Kubernetes Version: v1.15.9\n OpenEBS Version: 2.2.0\n\n**Node and BlockDevice details**: Attached three disks to three nodes(each node has one disk)\n\n Kubectl get nodes\n\n NAME STATUS ROLES AGE VERSION\n ip-192-168-29-151 Ready 16m v1.15.9\n ip-192-168-36-89 Ready 8h v1.15.9\n ip-192-168-74-129 Ready 8h v1.15.9\n\n Kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-36-89 10737418240 Claimed Active\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-74-129 10737418240 Claimed Active\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-29-151 10737418240 Claimed Active\n\n**CSPC Manifest**: Applied following CSPC manifest to provision cStor pools\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorPoolCluster\n metadata:\n name: cstor-disk-cspc\n namespace: openebs\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-74-129\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c2c846cce1befec7fbdcbae254329b0b\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-36-89\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-7d311a98255a454a717427b5c2d38426\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-29-151\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c608881cd3edbeab674a1aee7e0a1fc3\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n\nAfter applying the above CSPC manifest, the following three CStorPoolInstances(CSPI) were created.\n\n kubectl get cspi -n openebs\n\n NAME HOSTNAME FREE CAPACITY READONLY STATUS AGE\n cstor-disk-cspc-dvc2 ip-192-168-74-129 24100M 24111M false ONLINE 8h\n cstor-disk-cspc-f56z ip-192-168-36-89 24100M 24113200k false ONLINE 8h\n cstor-disk-cspc-q9yt ip-192-168-29-151 24100M 24113200k false ONLINE 8h\n\nNow everything looks good. After some time, the cluster has been scaled down **0** nodes and scaled back to **3** nodes. So after scaling operations following are new nodes in the cluster.\n\n Kubectl get nodes\n\n NAME STATUS ROLES AGE VERSION\n ip-192-168-14-90 Ready 118s v1.15.9\n ip-192-168-49-43 Ready 5m55s v1.15.9\n ip-192-168-94-113 Ready 4m6s v1.15.9\n\nAttached old disks that participated in pool creation to new nodes, and the following is blockdevice output.\n\n Kubectl get bd -n openebs\n\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-49-43 10737418240 Claimed Active\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-94-113 10737418240 Claimed Active\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-14-90 10737418240 Claimed Active\n\nFrom the above and previous output following are blockdevice mappings with zn old node and new node:\n\n Blockdevice Name Old Node New Node\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-36-89 ip-192-168-49-43\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-74-129 ip-192-168-94-113\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-29-151 ip-192-168-14-90\n\nOpenEBS **NodeDiskManager**(NDM) will automatically update the details in blockdevice CRs when the disks migrate to a new node. Based on the above output, update the CSPC manifest with new **nodeSelector** values.\n\n**Updated CSPC Manifest**:\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorPoolCluster\n metadata:\n name: cstor-disk-cspc\n namespace: openebs\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-94-113\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c2c846cce1befec7fbdcbae254329b0b\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-49-43\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-7d311a98255a454a717427b5c2d38426\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-14-90\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c608881cd3edbeab674a1aee7e0a1fc3\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n\nOnce the CSPC manifest is updated then CSPIs will automatically migrate to the new node (which can be verified using **kubectl get cspi -n openebs**).\n\n kubectl get cspi -n openebs\n\n NAME HOSTNAME FREE CAPACITY READONLY STATUS AGE\n cstor-disk-cspc-dvc2 ip-192-168-94-113 24100M 24111M false ONLINE 8h\n cstor-disk-cspc-f56z ip-192-168-49-43 24100M 24113200k false ONLINE 8h\n cstor-disk-cspc-q9yt ip-192-168-14-90 24100M 24113200k false ONLINE 8h\n\n**Note:** Along with CStorPoolInstance migration, CStorVolumeReplicas belongs to CSPI will also migrate automatically.\n","notHasFeatureImage":false,"slug":"migrate-cspis-to-a-different-node-by-moving-the-disks"},{"id":18,"title":"OpenEBS Backup/Restore for ZFS-LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"27-10-2020","tags":["OpenEBS"],"excerpt":"Overview of how to use Velero Backup/Restore plugin for ZFS-LocalPV to protect it against data loss.","content":"\n## Overview: OpenEBS Backup/Restore for ZFS-LocalPV\n\n**Backup** is the process of copying the data to a different/remote location to protect against accidental or corruption or any other type of data loss. Restore is the process of getting back the data from the backup. In this blog, I will discuss how we can use *Velero Backup/Restore* plugin for ***ZFS-LocalPV*** to protect it against data loss.\n\n### Pre-requisites\n\nWe should have installed the ZFS-LocalPV 1.0.0 or later version for Backup and Restore, see my previous[ blog](/blog/openebs-dynamic-volume-provisioning-on-zfs) for the steps to install the ZFS-LocalPV driver.\n\n### Setup\n\n**1.Install Velero CLI**\n\nDownload the 1.5 or later binary for ZFS-LocalPV. For Linux on amd64, we need to download below\n\n wget\n https://github.com/vmware-tanzu/velero/releases/download/v1.5.1/velero-v1.5.1-linux-amd64.tar.gz\n\nExtract the tarball:\n\n tar -xvf velero-v1.5.1-linux-amd64.tar.gz\n\nMove the extracted velero binary to somewhere in your $PATH (/usr/local/bin for most users).\n\nSee the detailed steps[ here](https://velero.io/docs/v1.5/basic-install/).\n\n**2.Deploy Velero**\n\nWe will be using minio for storage purpose in this blog, we need to setup the credential file first\n\n $ cat /home/pawan/velero/credentials-minio\n [default]\n aws_access_key_id = minio\n aws_secret_access_key = minio123\n\nWe can install Velero by using below command\n\n $ velero install --provider aws --bucket velero --secret-file /home/pawan/velero/credentials-minio --plugins velero/velero-plugin-for-aws:v1.0.0-beta.1 --backup-location-config region=minio,s3ForcePathStyle=\"true\",s3Url=http://minio.velero.svc:9000 --use-volume-snapshots=true --use-restic\n\nWe have to install the velero 1.5 or later version of velero for ZFS-LocalPV.\n\n**3.Deploy MinIO**\n\nDeploy the MinIO for storing the backup:-\n\n $ kubectl apply -f\n https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/sample/minio.yaml\n\nThe above MinIO uses tmp directory inside the pod to store the data for the demonstration purpose, so when restart happens, the backed up data will be gone. We should change the above YAML to use persistence storage to store the data when deploying it for the production.\n\nCheck the Velero Pods are UP and Running\n\n $ kubectl get po -n velero\n NAME READY STATUS RESTARTS AGE\n minio-d787f4bf7-xqmq5 1/1 Running 0 8s\n minio-setup-prln8 0/1 Completed 0 8s\n restic-4kx8l 1/1 Running 0 69s\n restic-g5zq9 1/1 Running 0 69s\n restic-k7k4s 1/1 Running 0 69s\n velero-7d9c448bc5-j424s 1/1 Running 3 69s\n\n**4.Setup OpenEBS Plugin**\n\nWe can Install the Velero Plugin for ZFS-LocalPV using the below command\n\n velero plugin add openebs/velero-plugin:2.2.0\n\nWe have to install the velero-plugin 2.2.0 or later version, which has the support for ZFS-LocalPV. Once the setup is done, we can go ahead and create the backup/restore.\n\n**5.Create the VSL**\n\nThe VSL(Volume Snapshot Location) has information about where the snapshot should be stored. To create the Backup/Restore, we can create the Volume Snapshot Location by applying the below YAML:\n\n apiVersion: velero.io/v1\n kind: VolumeSnapshotLocation\n metadata:\n name: default\n namespace: velero\n spec:\n provider: openebs.io/zfspv-blockstore\n config:\n bucket: velero\n prefix: zfs\n namespace: openebs # this is the namespace where ZFS-LocalPV creates all the CRs, passed as OPENEBS_NAMESPACE env in the ZFS-LocalPV deployment\n provider: aws\n region: minio\n s3ForcePathStyle: \"true\"\n s3Url: http://minio.velero.svc:9000\n\nHere, we have to provide the namespace, which we have used as OPENEBS_NAMESPACE env while deploying the ZFS-LocalPV. The ZFS-LocalPV Operator yamls uses “openebs” as the default value for OPENEBS_NAMESPACE env. Verify the volumesnapshot location:\n\n kubectl get volumesnapshotlocations.velero.io -n velero\n\n### Create Backup\n\nWe can use the below Velero command to create the backup:\n\n velero backup create my-backup --snapshot-volumes --include-namespaces= --volume-snapshot-locations=default --storage-location=default\n\nwe can add all the namespaces we want to be backed up in a comma-separated format in --include-namespaces parameter. We have to provide the VSL that we have created in --volume-snapshot-locations parameter.\n\nWe can check the backup status using the velero backup get command:\n\n $ velero backup get\n NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR\n my-backup InProgress 2020-09-14 21:09:06 +0530 IST 29d default \n\nThe status InProgress means that the backup is in progress. Wait for it to be Completed.\n\nWe can also create a scheduled backup which will take the backup periodically. For example, to take the full backup at every 5 min, we can create the below schedule :\n\n velero create schedule schd --schedule=\"*/5 * * * *\" --snapshot-volumes --include-namespaces=, --volume-snapshot-locations=default --storage-location=default\n\n### Restore\n\nIf the application and its PVC has been deployed in a namespace, then we can use the below Velero command to create the backup of the entire namespace:\n\n velero restore create --from-backup my-backup --restore-volumes=true --namespace-mappings :\n\nThe above command will create the backup of everything that is there in the namespace provided as --include-namespaces argument. We can provide the namespace mapping if we want to restore in a different namespace as --namespace-mappings parameter. If namespace mappings are not provided, it will restore in the source namespace only where the original pod and pvc was present. Now we can check the restore status:\n\n $ velero restore get\n NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR\n my-backup-20200914211331 my-backup InProgress 0 0 2020-09-14 21:13:31 +0530 IST \n\nOnce the Status is Completed, we can check the pods in the destination namespace and verify that everything is up and running. We can also verify that the data has been restored.\n\n### Summary\n\nAs demonstrated in this blog, OpenEBS makes it easy to take the backup of the Kubernetes applications, which we can use to Restore as part of disaster recovery. In my next blog, I will talk about how we can take the incremental backup of the volumes, which is space optimized backup for ZFS-LocalPV\n\n## Important links\n\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n[https://velero.io/docs/](https://velero.io/docs/v1.5/basic-install/)\n","notHasFeatureImage":false,"slug":"openebs-backup/restore-for-zfs-localpv"},{"id":19,"title":"OpenEBS 2.2.0 - Enhancements And New Storage Capabilities","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"20-10-2020","tags":["OpenEBS"],"excerpt":"OpenEBS 2.2.0 is here! Read this post to learn about the new updates.","content":"\n### **OpenEBS 2.2.0 is here**\n\nWe are excited to announce yet another ***OpenEBS*** release that comes with new storage capabilities, control plane enhancements, bug fixes, and new APIs for the world’s fastest storage engine built on RUST, also known as Mayastor.\n\nOpenEBS has seen a wider adoption among the users, thanks to the vibrant and growing community. Like in most of the OpenEBS releases, this release responds to the feedback received in the community. If you want to learn more about the project roadmap, please browse the following link:\n[https://github.com/openebs/openebs/blob/master/ROADMAP.md](https://github.com/openebs/openebs/blob/master/ROADMAP.md)\n\nIncremental Backup and Restore in ZFS local PV and pool and volume migration in cStor are the major release milestones made into the release. The pool migration in cStor solves the use-case of replacing a bad node with a new node or sending a node for maintenance on on-premise clusters. The migration feature provides great value in cloud-managed Kubernetes clusters, too, e.g., GKE, where node reboots or voluntary scale down of nodes can cause the disks to get removed. \n\nThis release is also special due to the [*Hacktoberfest*](https://hacktoberfest.digitalocean.com/) festival and would like to give a shout out to first-time contributors [@didier-durand](https://github.com/didier-durand), [@zlymeda](https://github.com/zlymeda), [@avats-dev](https://github.com/avats-dev), and many more.\n\n### **Key Highlights of OpenEBS 2.2.0 Release:**\n\n- Mayastor aims to be the world’s fastest container attached storage and is currently in alpha. The release introduced block device enumeration feature via the gRPC API and enhancement around storage pool finalizers.\n- ZFS local PV has become a popular storage engine built on local storage design and provides powerful storage features like snapshots and clones, raw block volume, etc. It also supports day two operations like volume resize and backup and restore via the pluggable Velero interface.\nSupport for Incremental Backup and Restore by enhancing the OpenEBS Velero Plugin has been a significant highlight for ZFS local PV release. \nTo learn more about this, please refer to the document [here](https://github.com/openebs/zfs-localpv/blob/master/docs/backup-restore.md).\n- OpenEBS Velero plugin connects the Velero interface to the OpenEBS storage engines to deliver backup/restore functionality. Velero Plugin has been enhanced to restore ZFS Local PV into a different cluster or a different node in the cluster and use custom certificates for S3 object storage.\n- CStor went into beta in 2.0 and has been enhanced to migrate the storage pool from one node to another node. This will help with scenarios where a Kubernetes node can be replaced with a new node but can be attached with the block devices from the old node that contain cStor Pool and the volume data.\n- OpenEBS node disk manager helps in block device discovery and management in a Kubernetes cluster and powers storage engines like cStor. Support to exclude multiple devices that could be mounted as host filesystem directories has been added.\nAn issue where NDM could cause data loss by creating a partition table on an uninitialized iSCSI volume has also been fixed.\n\n### **Useful Links and Summary:**\n\nIf you are interested in knowing more details regarding the changes that made to this release, please visit the release note [link](https://github.com/openebs/openebs/releases/tag/v2.2.0). To try out OpenEBS, you can visit [https://openebs.io/docs](/docs) and follow the user guides.\n\nYou can visit the following link to learn more or experiment with Mayastor\n[https://github.com/openebs/mayastor](https://github.com/openebs/mayastor)\n\nYou can visit the following link to learn more or experiment with ZFS local PV\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n\nTo learn more about the new cStor CSPC API, please visit the following link:\n[https://github.com/openebs/cstor-operators](https://github.com/openebs/cstor-operators)\n\nIf you have any feedback, questions, or suggestions — please reach out to the community on the #openebs channel in the Kubernetes workspace or consider opening a relevant issue at [Github](https://github.com/openebs/openebs).\n","notHasFeatureImage":false,"slug":"openebs-2.2.0---enhancements-and-new-storage-capabilities"},{"id":20,"title":"Scaling up cStor Volume Replica","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"07-10-2020","tags":["OpenEBS"],"excerpt":"OpenEBS provides volume replication through different storage engines. Learn how to scale up cStor Volume Replica.","content":"\nEven if a cluster is reliable, nodes can and do fail. Rebooting a node does not simulate a crash. There can be many reasons, such as catastrophic hardware failure, Operating System failure, or communication failure among the nodes. To overcome this hazardous situation, the Replication of volume becomes necessary.\n\nReplication is the process by which one or more volumes can be copied to maintain the significance of a cluster and to avoid data loss. OpenEBS provides volume replication through different storage engines. One of them is cStor Volume Replication.\n![Synchronous replication of data](https://lh5.googleusercontent.com/ijS24Ywabw-QkWWYbSLoOshGTi2SHZhdEFATaHIYbkNGK8lUq5SJrct6fNHfPjWcPTHGyvByS7uD1vYct2m5D6-HdRC2ZoMpS_c4Crw-9sREhPU-tXE8KAt-nWj7vYw99Ee_s1pE)\n#### Prerequisite for scaling up the replicas of cStor volume:\n\n- A cStor pool should be available, and replicas of this cStor volume should not be present on this cStor pool.\n- The OpenEBS version should be 1.3.0 or more.\n\n### Please follow the below steps for cStor Volume Replication:\n\nGet the StorageClass name using the following command:\n\n kubectl get sc\n\nExample Output:\n![](https://lh5.googleusercontent.com/lTma7ZqsAavmXEzGG_b4BXDMUEYXjFXf0xxnWgE70znfR_EzP3IorVFp0evkKoLMsBQ0D7gwOQxivB_bZxEcv2vhYZOe17k7mNyDBaPewTgiUdusrd3ow12ClBeQvZVmVzjDrdsI)\nThe storage class for cStor is ***openebs-sc-cstor***. Perform the following command to get the details of the corresponding StorageClass, which is used for creating the cStor volume :\n\n Kubectl get sc openebs-sc-cstor\n\nWe will get the Yaml file of the corresponding StorageClass ***openebs-sc-cstor***.\n![](https://lh5.googleusercontent.com/81DQJ-DhT3AKseMRfCZ4NpkmOPl2Tckm76jrUxE2eECY7lrejvNz3OjomFWmNiCRwm0L2seAWzmJJhe-8xcqFirBsEUedf2xzPN4NHq2RM2YYEZZv-iKpsE03j06EQi_D5kqnDCi)\nIn the Yaml above, We can see the Replica count Value is 1.\n\nGet the volume name using the following command:\n\n Kubectl get pvc\n\nGet the VOLUME name and use it in the following command to get the details of corresponding cStor volume. All commands are performed by considering the above PVC.\n\n kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=\n\nExample output:\n![](https://lh4.googleusercontent.com/FIOJchscq3lm7UJLwnk7i1oNne_RxhjIJzI3FMANxxkRhz4yWZAue-Wu1jD03ii2aMjtdDu3zr9C-0ZGaeazkvxb_JkGnxBBDza605w_p-v9BY1ER40f6DityHwimJvhvuAR8FcT)\nGet the details of existing cStor Volume Replica details using the following command:\n\n kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8\n\nExample output:\n![](https://lh3.googleusercontent.com/68NvgkfD7audTNZN1QLt6SVw4OvN_B3MIlnFnWm8MfgDziiexFX2qeI3tX6H1TCJJgrCA8b-nZQJoM6hx1QoYWOv4q74tKwB7nrZLc9xdluXRCvWTpj-sU6sIv7aJ0AMgL3rr1AR)\nPerform the following command to get complete details of the existing cStor volume replica:\n\n kubectl get cvr -n openebs -l openebs.io/persistent-volume=pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8\n\nGet the available cStor Pools for creating new cStor volume replica. The following command will get the other associated cStor pools details:\n\n kubectl get csp -l openebs.io/storage-pool-claim=cstor-disk-pool | grep -v cstor-disk-pool-hgt4\n\nExample Output:\n![](https://lh6.googleusercontent.com/lcbO830nSZgValr-I4ci7FHRa6Qvqf3eG-bycWHHAniRD8mb8dwRHOwxeVObFqj4FqvXbNkb_oZUdWhMgAQuHvU1pYDecvWXhDetYGdJADBQhWfzMuwJm4d9Ywgg6bAKkj-Sd79a)\nFrom the above example output, there are 2 cStor pools available, i.e., ***cstor-disk-pool-2phf*** and ***cstor-disk-pool-zm8l***. So it is possible to scale up the current volume replica count to 3 from 1. If there are no cStor pools available to perform volume replica scale-up, then follow the [steps](/docs/deprecated/spc-based-cstor#expanding-cStor-pool-to-a-new-node) to create a new cStor pool by updating existing SPC configuration.\n\nPerform the following command to get the details of the cStor Pool where new replica will be created:\n\n kubectl get csp -n openebs cstor-disk-pool-2phf -oyaml\n\nNote down following parameters from the output:\n\n- metadata.labels.cstorpool.openebs.io/name\n- metadata.labels.cstorpool.cstorpool.openebs.io/uid\n- metadata.annotations.cstorpool.openebs.io/hostname\n\nThe sample CVR Yaml is provided below:\n![](https://lh3.googleusercontent.com/JePqVqyIryf396SEkCf9NoS3kmPDXM0huqehkN3kX5f-eE7nX3-mCr42xriJeDKSNRgfVxSeQG_SUHkbqEZS4ktIzzcJ8VKCsFXuz4VhtdXpikLADE3eJdkgwH3zFd5PXRPfYc70)\nApply the updated CVR YAML spec to create the new replica of cStor volume using the following command:\n\n kubectl apply -f cvr.yaml\n\nExample Output:\n![](https://lh5.googleusercontent.com/JElB0d8zFXHoUh6wM0QpAshOmYbVXOvH5RIR9UjJ_svM67ZR2pq6cQ4ckrq0Qw6ACpRnOqO-6nUbvLUrDhFKvgZxjrh-ke0VHnKW-pR2oyzkgXdQuRATSwy9EVN19G458ZyR_9Xd)\nVerify if new CVR is created successfully using the following command:\n\n kubectl get cvr -n openebs\n\nExample output:\n![](https://lh4.googleusercontent.com/ql9j6Zcod6DT1vKhJrlJJaxk4YUN8Mf_o7LT3e-fBjjoybINByEwwDS5fln6K5BEJGW6vFfE8h2JA_2tFvQY5PQKo62eJvQfTE5j5JwECIz2oO3u_ypKHWRylL3gmU4KYlo4axtU)\nFrom the above output, a new replica of the cStor volume is created, and STATUS is showing as Offline.\n\nUpdate Desired Replication Factor in cStor volume with a new replica count. This can be updated by editing corresponding cStor volume CR YAML.\n\n kubectl edit cstorvolume pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8 -n openebs\n\nThe following is the snippet of updated cStor volume CR YAML:\n![](https://lh3.googleusercontent.com/lAisXwgequP2MyeCw1cVuwUYFG9G9L5U88olJ2CjbjIOpHjlMwn-K8p11ktaCjQfxK-u5EL-ebpZofD0W_LOKmfFa-wW3eTLtBpqSt7EPYvz5rQciYeaFdT6_7PCsJkdxPVHZCVg)\n\nIn the above snippet, the desiredReplicationFactor is updated to 2 from 1. Example output:\n![](https://lh6.googleusercontent.com/uBkJft958gfjATk070ZFZOMXaq7Sb1xnd5lBVMa2sKuXo-nxwrRxQS58TPgdpoLjMuMHvT4LwPscxPdT6kgwpaDVSraLmsNwWhfanMUrNVO72K8WgxwT3_or4EdzqQkWBgI-Ka84)\nVerify if the rebuilding has started on the new replica of the cStor volume. Once rebuilding has been completed, it will update its STATUS as Healthy. Get the latest status of the CVRs using the following command:\n\n kubectl get cvr -n openebs\n\nExample output:\n![](https://lh6.googleusercontent.com/1KjmeLgtvoFcBh0vVmB0iwj_gjo-Tkd3vVTTmaw3OaREY9KbvDUQLqyEu0Hj_aYKDpTIRSDVG2sOrTPMczJAPASlzFitSHDyocPV4Bb6IgajW-ArUpDKhi8StFesnHYZrUc3X9DJ)\n","notHasFeatureImage":false,"slug":"scaling-up-cstor-volume-replica"},{"id":21,"title":"Hacktoberfest 2020 - Contribute to OpenEBS","author":"MayaData Marketing","author_info":"Mayadata Marketing Team","date":"30-09-2020","tags":["OpenEBS"],"excerpt":"Hacktoberfest 2020 is almost here. Contribute to open source project, OpenEBS, to win exciting swag.","content":"\n### Hacktoberfest returns! Contribute to OpenEBS and win exciting swag\n\nThe seventh annual [***Hacktoberfest***](https://hacktoberfest.digitalocean.com/) celebration is almost here, and we at *OpenEBS* are happy to be participating in the contest once again. In August 2017, the OpenEBS community began growing and building a strong foundation for an open source project.\n\nWe were first introduced to *Hacktoberfest* by friends and peers at the DigitalOcean Bangalore Meetup and were immediately interested in participating. We enlisted OpenEBS as one of the projects participating in Hacktoberfest 2017. We were pleasantly surprised by the participation and enthusiasm that Hacktoberfest attracts from developers around the world. The PRs have been at their peak during Hacktoberfest.\n\n![](https://lh4.googleusercontent.com/Og_t8KLCiRni_LS66bpJsonSXMjcoAX671c8a2LD7ZjbkVdYZgZCRFq47sDC7hsEZt6qcaoCJPZi_gm2FnKmuzMvlg4UZAQKofU0agH2Z11TRmw6vBCQ8u3ssGfre75BN9OV-vOO)\n\n### Get started with OpenEBS this Hacktoberfest\n\nFollowing the smashing success we had when we participated in the event last few years, we’re going to do the same this year! MayaData makes it more exciting to participate in **Hacktoberfest** by running multiple Meetups throughout the month and helping contributors to get started with their favorite areas (in any of the programming languages) like website development and documentation enhancements. \n\nTo top it all, there are exciting prizes to be won and every contribution deserves an additional swag from MayaData. Read [this blog](https://blog.mayadata.io/openebs/experience-with-openebs-in-this-hacktoberfest) by Aswath K, one of last year’s weekly winners, who writes about his experience with OpenEBS in Hacktoberfest 2019.\n\n![](https://lh6.googleusercontent.com/2POqPppb7pyGM0OWwl_LlkHzwz-DSWXMMggxIeNCXvsU6EVVmNHdiIzIoTw23-ceK9R5iBleFMGiK-lw9JLtCP5VVjFGQS1QhIOXbpQhtvku5Gp5aCw4Eul_r6JcM-o0WuVZRZmj)\n\n### How do I contribute to OpenEBS?\n\nThat is an excellent question! OpenEBS is a Kubernetes native Container Attached Storage (CAS) that simplifies running Stateful workloads on Kubernetes. It is built on Microservices architectural patterns, fully automated development, and build pipelines.\n\nOpenEBS has several components that need your help from simple fixes like adding GitHub issue templates, to enhancing the components. These are developed using Go, Rust, Python, Javascript, Ansible, and many more interesting tools. OpenEBS is also a great way to start your journey into the exciting world of storage, containers, and Kubernetes.\n\nThe [architecture overview document ](https://github.com/openebs/openebs/blob/master/contribute/design/README.md) is a great place to start learning and picking up a component that speaks to your passion. You could start your first contribution by enhancing that document itself for providing more clarity.\n\nThere are many other [good first issues](https://github.com/search?q=org%3Aopenebs+is%3Aissue+label%3A%22good+first+issue%22) to pick from.\n\nContributions can be anything from creating issues, improving user and contributor documents, enhancing build and docker tools, fixing and enhancing code, or unit tests and e2e tests. If you are unsure where to start, begin a discussion with the contributors on [GitHub Discussions](https://github.com/openebs/openebs/discussions) or by joining [OpenEBS Community on Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F).\n\n### Will there be swag?\n\nYes. A big fat YES!\n\nThe official [Hacktoberfest](https://hacktoberfest.digitalocean.com/) will be giving away free t-shirts to every person making four pull requests to open source repositories during October, as well as limited-edition Hacktoberfest stickers to everyone who participates.\n\nOn top of this, you will also be able to get some exclusive and limited OpenEBS swag. When your PR to any OpenEBS repository is merged, we will contact you to fill out a form to send a special edition swag designed for Hacktoberfest.\n\nNot only this but by becoming a top weekly contributor, you’ll be able to grab even more swag.\n\nPrizes will be sent to quality contributions. The best PR will win a grand prize. Stay tuned to find out more.\n\nSo, what are you waiting for! Go get your git on and start contributing - we can’t wait to receive your PR.\n\nHappy hacking!\n\n### Getting Started:\n\n1. [https://hacktoberfest.digitalocean.com/](https://hacktoberfest.digitalocean.com/)\n2. Join [OpenEBS Community on Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F)\n3. Checkout the [OpenEBS Contributing guide](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md)\n4. Learn about the [architecture and components](https://github.com/openebs/openebs/blob/master/contribute/design/README.md) of OpenEBS\n5. Create new issues for your contribution or pick one of the existing issues from [https://github.com/openebs/openebs/issues](https://github.com/openebs/openebs/issues)\n","notHasFeatureImage":false,"slug":"hacktoberfest-2020---contribute-to-openebs"},{"id":22,"title":"OpenEBS StorageClasses For ZFS-LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"09-09-2020","tags":["OpenEBS","LocalPV","ZFS"],"excerpt":"In this blog, I will discuss various storage classes we can use to dynamically provision the volumes backed by ZFS-LocalPV Storage Pool.","content":"\nIn this blog, I will discuss various storage classes we can use to dynamically provision the volumes backed by ZFS Storage Pool.\n\nPlease read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the *ZFS-LocalPV*. Alternatively, you can also go through the [README](https://github.com/openebs/zfs-localpv/blob/master/README.md) section of the [*ZFS-LocalPV* repository](https://github.com/openebs/zfs-localpv).\n\n### **StorageClass Backed by ZFS Dataset**\n\nWe can create a StorageClass with the fstype as “zfs”. Here, the ZFS-LocalPV driver will create a ZFS dataset for the persistence storage. The application will get a dataset for the storage operation. We can also provide recordsize, compression, or dedup property in the StorageClass. The dataset will be created with all the properties mentioned in the StorageClass:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n recordsize: \"4k\"\n thinprovision: \"no\"\n fstype: \"zfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nWe have the thinprovision option as “no” in the StorageClass, which means that do reserve the space for all the volumes provisioned using this StorageClass. We can set it to “yes” if we don’t want to reserve the space for the provisioned volumes.\n\nThe allowVolumeExpansion is needed if we want to resize the volumes provisioned by the StorageClass. ZFS-LocalPV supports online volume resize, which means we don’t need to scale down the application. The new size will be visible to the application automatically.\n\nOnce the storageClass is created, we can go ahead and create the PVC and deploy a pod using that PVC.\n\n### **StorageClass Backed by ZFS Volume**\n\nThere are a few applications that need to have different filesystems to work optimally. For example, Concourse performs best using the “btrfs” filesystem ([https://github.com/openebs/zfs-localpv/issues/169](https://github.com/openebs/zfs-localpv/issues/169)). Here we can create a StorageClass with the desired fstype we want. The ZFS-LocalPV driver will create a ZVOL, which is a raw block device carved out from the mentioned ZPOOL and format it to the desired filesystem for the applications to use as persistence storage backed by ZFS Storage Pool:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: opeenbs-zfspv\n parameters:\n volblocksize: \"4k\"\n thinprovision: \"yes\"\n fstype: \"btrfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nHere, we can mention any fstype we want. As of 0.9 release, the driver supports ext2/3/4, xfs, and btrfs fstypes for which it will create a ZFS Volume. Please note here, if fstype is not provided in the StorageClass, the k8s takes “ext4\" as the default fstype. Here also we can provide volblocksize, compression, and dedup properties to create the volume, and the driver will create the volume with all the properties provided in the StorageClass.\n\nWe have the thinprovision option as “yes” in the StorageClass, which means that it does not reserve the space for all the volumes provisioned using this StorageClass. We can set it to “no” if we want to reserve the space for the provisioned volumes.\n\n### **StorageClass for Sharing the Persistence Volumes**\n\nBy default, the ZFS-LocalPV driver does not allow Volumes to be mounted by more than one pod. Even if we try to do that, only one Pod will come into the running state, and the other Pod will be in ContainerCreating state, and it will be failing on the mount.\n\nIf we want to share a volume among multiple pods, we can create a StorageClass with the “shared” option as “yes”. For this, we can create a StorageClass backed by ZFS dataset as below :\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"zfs\"\n shared: \"yes\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nOr, we can create the StorageClass backed by ZFS Volume for sharing it among multiple pods as below :\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"ext4\"\n shared: \"yes\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nHere, we have to note that all the Pods using that volume will come to the same node as the data is available on that particular node only. Also, applications need to be aware that the volume is shared by multiple pods and should synchronize with the other Pods to access the data from the volume.\n\n### **StorageClass With k8s Scheduler**\n\nThe ZFS-LocalPV Driver has its own scheduling logic, where it creates the volume where the ZFS Pool is less loaded with the volumes. Here, it just checks the volume count and creates the volume where less volume is configured in a given ZFS Pool. It does not account for other factors like available CPU or memory while making scheduling decisions. So if you want to use node selector/affinity rules on the application pod or have CPU/Memory constraints, the Kubernetes scheduler should be used. To make use of Kubernetes scheduler, we can set the volumeBindingMode as WaitForFirstConsumer in the storage class:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"zfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n volumeBindingMode: WaitForFirstConsumer\n\nHere, in this case, the Kubernetes scheduler will select a node for the POD and then ask the ZFS-LocalPV driver to create the volume on the selected node. The driver will create the volume where the POD has been scheduled.\n\n### **StorageClass With Custom Node Labels**\n\nThere can be a use case where we have certain kinds of ZFS Pool present on certain nodes only, and we want a particular type of application to use that ZFS Pool. We can create a storage class with `allowedTopologies` and mention all the nodes there where that pool is present:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: nvme-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: openebs.io/nodename\n values:\n - node-1\n - node-2\n\nHere we can have ZFS Pool of name “zfspv-pool” created on the nvme disks and want to use this high performing ZFS Pool for the applications that need higher IOPS. We can use the above StorageClass to create the PVC and deploy the application using that.\n\nThe ZFS-LocalPV driver will create the Volume in the Pool “zfspv-pool” present on the node with fewer of volumes provisioned among the given node list. In the above StorageClass, if there provisioned volumes on node-1 are less, it will create the volume on node-1 only. Alternatively, we can use `volumeBindingMode: WaitForFirstConsumer` to let the k8s select the node where the volume should be provisioned.\n\nThe problem with the above StorageClass is that it works fine if the number of nodes is less, but if the number of nodes is huge, it is cumbersome to list all the nodes like this. In that case, what we can do is, we can label all the similar nodes using the same key value and use that label to create the StorageClass.\n\n pawan@pawan-master:~/pawan$ kubectl label node pawan-node-2 openebs.io/zpool=nvme\n node/pawan-node-2 labeled\n pawan@pawan-master:~/pawan$ kubectl label node pawan-node-1 openebs.io/zpool=nvme\n node/pawan-node-1 labeled\n\nNow, restart the ZFS-LocalPV Driver (if already deployed otherwise, please ignore) so that it can pick the new node label as the supported topology.\n\n $ kubectl delete po -n kube-system -l role=openebs-zfs\n\nNow, we can create the StorageClass like this:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: nvme-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: openebs.io/zpool\n values:\n - nvme\n\nHere, the volumes will be provisioned on the nodes which has label “openebs.io/zpool” set as “nvme”.\n\n### **Conclusion :**\n\nWe can set up different kinds of StorageClasses as per our need, and then we can proceed with PVC and POD creation. The driver will take the care of honoring the requests put in the PVC and the StorageClass.\n\nI hope you found this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"openebs-storageclasses-for-zfs-localpv"},{"id":23,"title":"Handling node down/cluster scaling on ZFS LocalPV backed workloads","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"01-09-2020","tags":["Kubernetes"],"excerpt":"Step-by-step blog on how MySQL app deployment runs on OpenEBS ZFS LocalPV device, handled when a node down/cluster scale down situation happens in GKE cluster","content":"\nKubernetes is increasingly used for running production-grade stateful services. Organizations are making progress on a containerized form of their production workloads for running in Kubernetes. There are already solutions available for the containerized version of stateful applications, network, storage, etc.\n\nAs everyone knows, OpenEBS is one of the leading containerized storage solutions for Kubernetes, and it is a growing Sandbox project in CNCF. MayaData is the primary maintainer and contributor of OpenEBS along with other companies. MayaData also contributed another open source project, Litmus, into CNCF, which does mostly Chaos engineering in Kubernetes, which helps SREs and developers to do all kinds of testing of their applications and components in Kubernetes before going into Production.\n\nA persistent storage solution for running any stateful application is a must requirement, be it a **Deployment** or **StatefulSet**. OpenEBS provides various storage engines, and each storage engine is suitable for specific applications or workloads. Some engines provide storage level synchronous replication, capable of taking snapshots and cloning, backup and restore, volume expansion, CSI complaint, performance-oriented, etc. So choosing the engine based on the workload requirement is an important activity.\n\nOpenEBS provides dynamic provisioning of ***ZFS LocalPV*** using external device/devices. OpenEBS ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. Using a ***ZFS Local PV*** has the following advantages   as opposed to Kubernetes native Local PV backed by direct-attached devices:\n\n- Sharing of the devices among multiple application pods.\n- Enforcing quota on the volumes makes sure the pods don’t consume more than their capacity.\n- Ability to take snapshots of the LocalPV\n- Ability to sustain the disk failures — using the ZPOOL RAID functionality\n- Ability to use data services like compression and encryption.\n- Ability to resize the PV capacity.\n\nIn this article, we provisioned a MySQL deployment on an ***OpenEBS ZFS LocalPV*** device dynamically.\n\nThis article is a step-by-step instruction. We will mention how a MySQL application deployment running on the OpenEBS ZFS LocalPV device volume is getting handled when a Node down scenario or a cluster scale down situation happens in the GKE cluster. In GKE and some other managed clusters like EKS, the node name will change if the cluster undergoes a scale down and scale up operation has performed. So the application running on the OpenEBS ZFS LocalPV device will not be able to attach to the new node since the corresponding PV and ZFS Volume has a binding of volume node affinity. We need to update the new node name details in PV and ZFS volume, where the disk got attached.\n\nIn this article, we are discussing the steps that need to be performed to make the application into a running state when a Node down / scale down cluster scenario has happened. This situation is usually required in case of Managed clusters where the Node name will get changed during this scenario. As stated earlier, the following approach works fine for both Deployment type and StatefulSet type, but ensure that the next steps are correctly performed. Let’s start with the step by step instructions once you have scaled up the cluster after a scale down scenario.\n\n1. Verify all nodes are now in Ready state.\n ```\n $ kubectl get node\n NAME STATUS ROLES AGE VERSION\n gke-openebs-mysql-default-pool-dd23ce6b-f8rd Ready 24m v1.16.13-gke.1\n gke-openebs-mysql-default-pool-dd23ce6b-lwr3 Ready 24m v1.16.13-gke.1\n gke-openebs-mysql-default-pool-dd23ce6b-zzqx Ready 24m v1.16.13-gke.1\n ```\n2. Label all the nodes with the same custom label used in the nodeSelector field in the STS app. In my case, there is no custom node label used in application deployment. So we are skipping this step.\n\n3. Attach the disk randomly to any node in the same zone.\n \n ```\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-j894 --disk mysql-disk1 --device-name mysql-disk1 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-prv2 --disk mysql-disk2 --device-name mysql-disk2 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-tf5j --disk mysql-disk3 --device-name mysql-disk3 --zone=us-central1-c\n ```\n4. Ensure that ZFS utils packages are installed on your worker nodes. If it is not installed, ZFS packages can be installed.\n \n ```\n $ sudo su -\n $ sudo apt-get update\n $ apt-get install zfsutils-linux -y\n $ zpool import zfspv-pool\n ```\n \n Verify Zpool information.\n \n ```\n $ zpool list\n NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT\n zfspv-pool 14.5G 463M 14.0G - 0% 3% 1.0\n ```\n5. Verify the details of the ZFS dataset detail where the volume is present. This information is required in step 10. In the case of Statefulset, the ZFS dataset will be present on multiple nodes. So you should note down the details of the ZFS dataset and corresponding node information.\n \n ```\n $ sudo su\n $ root@gke-openebs-mysql-default-pool-dd23ce6b-tf5j:~# zfs list\n NAME USED AVAIL REFER MOUNTPOINT\n zfspv-pool 10.6G 3.42G 96K /zfspv-pool\n zfspv-pool/pvc-e299a9db-0903-417b-8034-03c3dc77af87 10.6G 13.6G 462M -\n ```\n\n From the above information, the ZFS dataset is present on Node `gke-openebs-mysql-default-pool-dd23ce6b-tf5j`. We have to update this node information in the nodeSelector field in step 10 and as `ownerNodeID` in step 11.\n\n6. Ensure OpenEBS ZFS driver pods are running in the `kube-system` namespace.\n \n ```\n $ kubectl get pods -n kube-system -l role=openebs-zfs\n NAME READY STATUS RESTARTS AGE\n openebs-zfs-controller-0 5/5 Running 0 91m\n openebs-zfs-node-29dlp 2/2 Running 0 14m\n openebs-zfs-node-bssq7 2/2 Running 0 14m\n openebs-zfs-node-p54tq 2/2 Running\t 0 14m\n ```\n7. Check the status of the application pod. It will be in the `Pending` state.\n \n ```\n $ kubectl get pod\n NAME READY STATUS RESTARTS AGE\n percona-9fbdb8678-z79vr 0/1 Pending 0 70m\n ```\n8. Get the PV details of the associated application\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO Delete Bound default/demo-vol1-claim openebs-zfspv 33m\n ```\n9. Create a directory and copy the YAML spec of all the associated PVs into it like below.\n \n ```\n $ mkdir mysql-restore\n $ cd mysql-restore/\n $ kubectl get pv pvc-e299a9db-0903-417b-8034-03c3dc77af87 -o yaml --export > pv1.yaml\n ```\n \n Note: If it is StatefulSet, take the YAML spec of all the associated PVs of that application.\n\n10. Modify the above-copied YAML with the new hostname in the copied YAML of PV. The following is a snippet of PV spec where it mentions the new node name where the ZFS volume is created. The node information will be obtained from step 5.\n ```\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: openebs.io/nodename\n operator: In\n values:\n - gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n persistentVolumeReclaimPolicy: Delete\n storageClassName: openebs-zfspv\n ```\n11. Also, update the node information where the ZFS dataset resides into `zv`(ZFS volume) cr. The node name has to be given to the path  `spec.ownerNodeID`.\n \n ```\n $ kubectl edit zv pvc-e299a9db-0903-417b-8034-03c3dc77af87 -n openebs\n ```\n \n The following is the snippet of the modified information.\n \n ```\n spec:\n capacity: \"10737418240\"\n compression: \"off\"\n dedup: \"off\"\n fsType: ext4\n ownerNodeID: gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n poolName: zfspv-pool\n ```\n12. Now get the PV and then delete the PV\n \n ```\n spec:\n capacity: \"10737418240\"\n compression: \"off\"\n dedup: \"off\"\n fsType: ext4\n ownerNodeID: gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n poolName: zfspv-pool\n ```\n ```\n $ kubectl delete pv pvc-e299a9db-0903-417b-8034-03c3dc77af87\n persistentvolume \"pvc-e299a9db-0903-417b-8034-03c3dc77af87\" deleted\n ```\n \n The deletion of the PV will not be completed since it has the finaliser set with the PV. So we need to cancel the ongoing operation and then edit the PV and remove Finalizers. Once finalizers are removed, the volume will be automatically deleted.\n\n13. Verify that the PV of the application has been removed successfully.\n \n ```\n $ kubectl delete pv pvc-e299a9db-0903-417b-8034-03c3dc77af87\n persistentvolume \"pvc-e299a9db-0903-417b-8034-03c3dc77af87\" deleted\n ```\n14. Now, apply the updated YAML files of the PV. \n \n ```\n $ kubectl apply -f pv1.yaml \n ```\n \n Note: Perform the same for other PVs as well if the application is a StatefulSet.\n\n15. Verify that if PODs are started `Running` from `Pending` state.\n \n ```\n $ kubectl get pod -o wide\n NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\n percona-9fbdb8678-z79vr 1/1 Running 0 6h43m 10.4.2.2 gke-openebs-mysql-default-pool-dd23ce6b-tf5j \n ```\n \n Verify the PV, PVC, and ZV associated with the MySQL application.\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO Delete Bound default/demo-vol1-claim openebs-zfspv 150m\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-vol1-claim Bound pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO openebs-zfspv 7h7m\n\n $ kubectl get zv -n openebs\n NAME ZPOOL NODE SIZE STATUS FILESYSTEM AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 zfspv-pool gke-openebs-mysql-default-pool-dd23ce6b-tf5j 10737418240 Ready ext4 7h7m\n\n16. Login to the application and verify that you are able to access the data.\n ```\n $ kubectl exec -it percona-9fbdb8678-z79vr sh\n sh-4.2$ mysql -uroot -pk8sDem0;\n mysql: [Warning] Using a password on the command line interface can be insecure.\n Welcome to the MySQL monitor. Commands end with ; or \\g.\n Your MySQL connection id is 2\n Server version: 5.7.30-33 Percona Server (GPL), Release 33, Revision 6517692\n \n Copyright (c) 2009-2020 Percona LLC and/or its affiliates\n Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.\n \n Oracle is a registered trademark of Oracle Corporation and/or its\n affiliates. Other names may be trademarks of their respective\n owners.\n \n Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n \n mysql> SHOW DATABASES;\n +--------------------+\n | Database |\n +--------------------+\n | information_schema |\n | mysql |\n | performance_schema |\n | pets |\n | sys |\n +--------------------+\n 5 rows in set (0.11 sec)\n \n mysql> USE pets;\n Reading table information for completion of table and column names\n You can turn off this feature to get a quicker startup with -A\n \n Database changed\n mysql> SELECT * FROM cats;\n +----+---------+--------+------------+\n | id | name | owner | birth |\n +----+---------+--------+------------+\n | 1 | Sandy | Lennon | 2015-01-03 |\n | 2 | Cookie | Casey | 2013-11-13 |\n | 3 | Charlie | River | 2016-05-21 |\n +----+---------+--------+------------+\n 3 rows in set (0.00 sec)\n ```","notHasFeatureImage":false,"slug":"handling-node-down/cluster-scaling-on-zfs-localpv-backed-workloads"},{"id":24,"title":"Recover from Volume Multi Attach Error in On-Prem Kubernetes Clusters","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. OpenSource Enthusiast","date":"27-08-2020","tags":["Kubernetes"],"excerpt":"In this blog, we'll talk about recovering from volume multi attach error in On-Prem Kubernetes clusters.","content":"\nIf you have an unmanaged Kubernetes Cluster that you have deployed on-prem or on cloud, you would have noticed that your Stateful Application pods error out with Multi-attach error whenever the node running the stateful application is abruptly shut down.\n\nThis has been a long outstanding issue in Kubernetes and is being actively worked on. Please refer to the following Kubernetes issues:\n\n- [https://github.com/kubernetes/enhancements/pull/1116](https://github.com/kubernetes/enhancements/pull/1116)\n- [https://github.com/kubernetes/kubernetes/issues/86281](https://github.com/kubernetes/kubernetes/issues/86281)\n- [https://github.com/kubernetes/kubernetes/issues/53059#issuecomment-619428689](https://github.com/kubernetes/kubernetes/issues/53059#issuecomment-619428689)\n\nThe main reason for this issue being hard to resolve is that there is no right way to determine if the node is really shut down or if it is due to a network/split-brain condition to the master nodes. And it gets a little harder with Stateful applications as we need to really determine that data is written down from older nodes to the disks, before forcibly remounting onto the new node.\n\nIn this blog, I will provide an alternate approach that can be used to work around this issue and bring your applications back online. The solution is loosely based on the same approach that Managed Kubernetes clusters like GKE/EKS perform to handle this scenario. The managed clusters use out-of-band communication to determine if the node is shut down and delete the node resources.\n\nI will demonstrate the approach of removing the node resource as a safe way to recover volumes using the following example.\n\n## **Problem**\n\n1. ### Start with a Stateful application:\n I have a three node cluster with k8s version 1.15.3, to reproduce the Volume multi-attach error scenario. Deployed OpenEBS version 1.3, using cstor csi based volume and mounted to Percona pod scheduled in node csi-node2.mayalabs.io.\n ```\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node2.mayalabs.io Ready none 39d v1.15.3\n csi-node3.mayalabs.io Ready none 39d v1.15.3\n\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-csi-vol-claim Bound pvc-b39248ab-5a99-439b-ad6f-780aae30626c 10Gi RWO openebs-csi-cstor-sparse 72m\n\n\n $ kubectl get pods -owide\n NAME READY STATUS RESTARTS AGE NODE\n percona-6795d6fb68-pqvqh 1/1 Running 0 66m csi-node2.mayalabs.io\n ```\n\n VolumeAttachment resource created only for volume attached to Node2 in case of CSI based persistent volumes\n ```\n $ kubectl get volumeattachment\n NAME ATTACHER PV NODE ATTACHED AGE\n csi-9f7704015b456f146ce8c6c3bd80a5ec6cc55f4f5bfb90c61c250d0b050a283c openebs-csi.openebs.io pvc-b39248ab-5a99-439b-ad6f-780aae30626c csi-node2.mayalabs.io true 66m\n ```\n\n2. ### Node ShutDown:\n\n Shutting down kubelet service in Node2 as Percona application pod has been scheduled here, to make node `NotReady` state in the Kubernetes cluster.\n\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node2.mayalabs.io NotReady none 5m v1.15.3\n csi-node3.mayalabs.io Ready none 37d v1.15.3\n \n\n In this case, the Percona pod will get stuck in a container creating a state with a multi-attach error.\n\n## **Solution:**\n\nAlthough these solutions are generic to recover the volume from the multi attach error. Based on some extra steps, I have divided them into two different sanctions based on the type of Kubernetes volumes. One is dynamic in-tree or external Kubernetes volumes, an older way of provisioning Kubernetes volumes, and the other one is CSI based Kubernetes volumes.\n\n### **Dynamic/Static Provisioner Based Volumes:**\n\n#### **Deleting Node Resource:**\n\nHere after deleting the Node resource below node related events will be generated which will trigger the force deletion of the pods, and they apparently will be scheduled to other available nodes:\n\n $ kubectl delete nodes csi-node2.mayalabs.io\n node \"csi-node2.mayalabs.io\" deleted\n \n\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node3.mayalabs.io Ready none 37d v1.15.3\n \n\n##### **Check the kube-controller logs for events:**\n\n I1021 10:37:44.336523 1 attach_detach_controller.go:573] error removing node \"csi-node2.mayalabs.io\" from desired-state-of-world: failed to delete node \"csi-node2.mayalabs.io\" from list of nodes managed by attach/detach controller--the node still contains 1 volumes in its list of volumes to attach\n I1021 10:37:45.003243 1 event.go:258] Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"csi-node2.mayalabs.io\", UID:\"30ca0f5e-3a8f-4d0f-99fc-776d7051fd3d\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RemovingNode' Node csi-node2.mayalabs.io event: Removing Node csi-node2.mayalabs.io from Controller\n \n \n I1021 10:37:55.273070 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/openebs-csi-node-cjzlz\n I1021 10:37:55.318908 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/openebs-csi-node-cjzlz succeeded\n I1021 10:37:55.318979 1 gc_controller.go:62] PodGC is force deleting Pod: openebs/openebs-ndm-8ntsv\n I1021 10:37:55.352796 1 gc_controller.go:166] Forced deletion of orphaned Pod openebs/openebs-ndm-8ntsv succeeded\n I1021 10:37:55.354071 1 gc_controller.go:62] PodGC is force deleting Pod: openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\n I1021 10:37:55.420779 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"openebs\", Name:\"cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\", UID:\"\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'TaintManagerEviction' Cancelling deletion of Pod openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\n I1021 10:37:55.442403 1 gc_controller.go:166] Forced deletion of orphaned Pod openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn succeeded\n I1021 10:37:55.442568 1 gc_controller.go:62] PodGC is force deleting Pod: default/percona-6795d6fb68-b7dvl\n I1021 10:37:55.446368 1 event.go:258] Event(v1.ObjectReference{Kind:\"ReplicaSet\", Namespace:\"openebs\", Name:\"cspc-sparse-disk-pool-gg82-d9b4bff4d\", UID:\"04f87ed3-c401-4688-8691-0716dc4693fe\", APIVersion:\"apps/v1\", ResourceVersion:\"7063677\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: cspc-sparse-disk-pool-gg82-d9b4bff4d-q2l2b\n I1021 10:37:55.541929 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-b7dvl\", UID:\"\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'TaintManagerEviction' Cancelling deletion of Pod default/percona-6795d6fb68-b7dvl\n I1021 10:37:55.599155 1 gc_controller.go:166] Forced deletion of orphaned Pod default/percona-6795d6fb68-b7dvl succeeded\n I1021 10:37:55.599224 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/kube-proxy-b9q25\n I1021 10:37:55.613517 1 event.go:258] Event(v1.ObjectReference{Kind:\"ReplicaSet\", Namespace:\"default\", Name:\"percona-6795d6fb68\", UID:\"50b82272-9874-4688-8362-7c759ae63aef\", APIVersion:\"apps/v1\", ResourceVersion:\"7063669\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: percona-6795d6fb68-pqvqh\n W1021 10:37:55.621461 1 reconciler.go:328] Multi-Attach error for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" Volume is already exclusively attached to node csi-node2.mayalabs.io and can't be attached to another\n I1021 10:37:55.629191 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-pqvqh\", UID:\"26ee6109-9d7e-4729-843b-18bb88926c87\", APIVersion:\"v1\", ResourceVersion:\"7063944\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" Volume is already exclusively attached to one node and can't be attached to another\n I1021 10:37:55.742498 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/kube-proxy-b9q25 succeeded\n I1021 10:37:55.742697 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/calico-node-4fv7n\n I1021 10:37:55.787435 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/calico-node-4fv7n succeeded\n \n\n### **CSI Based Volumes:**\n\n#### **Attach-Detach Controller:**\n\nIf the volume is created using CSI Provisioner, a custom resource `volumeattachment` would be created. The Attach-detach controller will wait for the `maxWaitForUnmountDuration` i.e., 6 minutes to forcefully detach the attached volume from the node. Then the CR will be recreated and attach to any available node. To recover from the multi-attach error, this `volumeattachment` CR can be deleted along with node CR. Therefore the time taken to mount the volume on a new node will be reduced by 6 minutes.\n\n $ kubectl get volumeattachment pvc-b39248ab-5a99-439b-ad6f-780aae30626c\n NAME ATTACHER PV NODE ATTACHED AGE\n csi-6ae3ead0d1c3a6e73e7d4c8e27f9098b927f3e4edc21bcf6bb7cf3fcdb4101de openebs-csi.openebs.io pvc-b39248ab-5a99-439b-ad6f-780aae30626c csi-node1.mayalabs.io true 11m\n \n\n \n $ kubectl logs -f kube-controller-manager-csi-master.mayalabs.io -n kube-system\n\n \n ```\n W1021 10:43:55.629673 1 reconciler.go:232] attacherDetacher.DetachVolume started for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") on node \"csi-node2.mayalabs.io\" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching\n I1021 10:43:55.671000 1 operation_generator.go:526] DetachVolume.Detach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") on node \"csi-node2.mayalabs.io\" \n I1021 10:43:55.730390 1 reconciler.go:288] attacherDetacher.AttachVolume started for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" \n I1021 10:43:55.752273 1 operation_generator.go:358] AttachVolume.Attach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" \n I1021 10:43:55.753344 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-pqvqh\", UID:\"26ee6109-9d7e-4729-843b-18bb88926c87\", APIVersion:\"v1\", ResourceVersion:\"7063944\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulAttachVolume' AttachVolume.Attach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" \n ```\n \n\nThat's it for today's post. I hope you find it helpful. Feedback and comments are appreciated.\n","notHasFeatureImage":false,"slug":"recover-from-volume-multi-attach-error-in-on-prem-kubernetes-clusters"},{"id":25,"title":"Resize Kubernetes StatefulSets without impact","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, Passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"25-08-2020","tags":["Kubernetes","OpenEBS"],"excerpt":"Read this post if you are a cStor CSI user who's looking to resize statefulsets without any impact of StatefulSet applcations. ","content":"\nIn large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. MayaData, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. cStor is one of the storage engines of OpenEBS.\n\nThis blog is for OpenEBS users, specifically cStor CSI users looking to resize their ***Kubernetes StatefulSets*** without any impact of StatefulSet applications.\n\n### **Steps involved to resize** ***Kubernetes StatefulSets***\n\n1. Increase the volume size of the PVC.\n\n2. Update StatefulSet volumeClaimTemplate storage capacity.\n\n### Infrastructure details:\n\n Kubernetes Cluster: Bare Metal\n Kubernetes Version: v1.17.2\n OpenEBS Version: 2.0.0\n\n### Installation of CStor setup:\n\nApplied OpenEBS 2.0.0 version of cStor operator [yaml](https://github.com/openebs/charts/blob/gh-pages/2.0.0/cstor-operator.yaml) and ndm operator [yaml](https://github.com/openebs/charts/blob/gh-pages/2.0.0/ndm-operator.yaml) via kubectl apply and provisioned cStor pools using CSPC API by following the steps mentioned [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md).\n\n system@master$ kubectl get cspc -n openebs\n NAME HEALTHYINSTANCES PROVISIONEDINSTANCES DESIREDINSTANCES AGE\n cspc-stripe 3 3 3 3m23s\n\n### Create StorageClass:\n\nCreate a StorageClass pointing to the above cStor pool(cspc-stripe) which will help in provisioning the cStor volume. For expanding the volumes **allowVolumeExpansion** parameter should set to true\n\n system@master$ cat csi-sc.yaml \n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-csi-sc\n provisioner: cstor.csi.openebs.io\n allowVolumeExpansion: true\n parameters:\n cas-type: cstor\n replicaCount: \"1\"\n cstorPoolCluster: cspc-stripe\n\n### Provision StatefulSet:\n\nCreate StatefulSet to point to the above storageclass. In this example, mongo statefulset will be provisioned and volume size will be expanded from 5Gi to 15Gi. Below is the sample statefulset used for expansion:\n\n system@master$ cat mongo-sts.yaml \n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: mongo\n Namespace: mongo-ns\n spec:\n selector:\n matchLabels:\n role: mongo\n environment: test\n serviceName: \"mongo\"\n replicas: 3\n template:\n metadata:\n labels:\n role: mongo\n environment: test\n spec:\n terminationGracePeriodSeconds: 10\n containers:\n - name: mongo\n image: mongo:latest\n imagePullPolicy: IfNotPresent\n command:\n - mongod\n - \"--replSet\"\n - rs0\n ports:\n - containerPort: 27017\n volumeMounts:\n - name: mongo-persistent-storage\n mountPath: /data/db\n - name: mongo-sidecar\n image: cvallance/mongo-k8s-sidecar\n ...\n ...\n volumeClaimTemplates:\n - metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: \"openebs-csi-sc\"\n accessModes: [\"ReadWriteOnce\"]\n resources:\n requests:\n storage: 5Gi\n\nAfter applying the above YAML. This results in the provisioning of volumes with **5Gi** capacity as mentioned in **volumeClaimTemplates,** and applications are in Running state.\n\n system@master$ kubectl get pods -n mongo-ns\n NAME READY STATUS RESTARTS AGE\n mongo-0 2/2 Running 0 8m24s\n mongo-1 2/2 Running 0 7m5s\n mongo-2 2/2 Running 0 6m4s\n \n -----------------------------------------------------------------------------------\n system@master$ kubectl get pvc -n mongo-ns\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n mongo-persistent-storage-mongo-0 Bound pvc-4926e6bb-3226-4de9-add5-e603ea2b948b 5Gi RWO openebs-csi-sc 6m48s\n mongo-persistent-storage-mongo-1 Bound pvc-fcde1fde-65ee-48ab-8f79-b7c68edfb934 5Gi RWO openebs-csi-sc 5m29s\n mongo-persistent-storage-mongo-2 Bound pvc-d94e5304-7f29-4d6c-b157-1e65f75511f1 5Gi RWO openebs-csi-sc 4m28s\n\nNow verify the size of the volume by exec into any one of the application pods. In below output **/dev/sde** is the persistent volume and it’s capacity is **5G**.\n\n system@master$ kubectl exec -it mongo-0 -n mongo-ns -c mongo -- df -h\n Filesystem Size Used Avail Use% Mounted on\n overlay 98G 12G 82G 13% /\n tmpfs 64M 0 64M 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sde 4.8G 311M 4.5G 7% /data/db\n /dev/sda1 98G 12G 82G 13% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /proc/acpi\n tmpfs 3.9G 0 3.9G 0% /proc/scsi\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nNow, let’s resize the volume capacity by following the steps mentioned in the blog's beginning.\n\nStep1: Increase the volume size of PVC\n\nExpand the size of the PVC size by applying below command on all the StatefulSet volumes:\n\n kubectl patch pvc mongo-persistent-storage-mongo-0 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n \n kubectl patch pvc mongo-persistent-storage-mongo-1 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n \n \n kubectl patch pvc mongo-persistent-storage-mongo-2 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n\nAfter patching the above PVCs **openebs-cstor-csi** plugin is responsible for performing the resize operation on cStor volume. Verify whether volumes are expanded successfully by performing the following commands\n\n system@master: kubectl get pvc -n mongo-ns\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n mongo-persistent-storage-mongo-0 Bound pvc-4926e6bb-3226-4de9-add5-e603ea2b948b 15Gi RWO openebs-csi-sc 32m\n mongo-persistent-storage-mongo-1 Bound pvc-fcde1fde-65ee-48ab-8f79-b7c68edfb934 15Gi RWO openebs-csi-sc 30m\n mongo-persistent-storage-mongo-2 Bound pvc-d94e5304-7f29-4d6c-b157-1e65f75511f1 15Gi RWO openebs-csi-sc 29m\n\nAbove PVC shows that volumes are expanded successfully. Now verify capacity by exec in to any one of the application pods\n\n system@master$ kubectl exec -it mongo-0 -n mongo-ns -c mongo -- df -h\n Filesystem Size Used Avail Use% Mounted on\n overlay 98G 12G 82G 13% /\n tmpfs 64M 0 64M 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sde 15G 313M 15G 3% /data/db\n /dev/sda1 98G 12G 82G 13% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /proc/acpi\n tmpfs 3.9G 0 3.9G 0% /proc/scsi\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nNote: This step only helps to resize the volumes that are already provisioned for consuming via statefulsets. If StatefulSet is scaled up, then the newly provisioning volume will have the old size(since the volumeClaimTemplate is not yet updated in the application spec).\n\n**Step2: Update StatefulSet volumeClaimTemplate storage size**\n\nSince natively resizing of statefulset volumeClaimTemplates is not yet supported in Kubernetes, one needs to follow this step to update size in the volume claim template. For further updates, track the enhancement proposal [here](https://github.com/kubernetes/enhancements/pull/1848).\n\nUpdate the capacity in volumeClaimTemplate of applied statefulset yaml lke below example:\n\n system@master$ cat mongo-sts.yaml\n cat mongo-sts.yaml \n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: mongo\n namespace: mongo-ns\n spec:\n selector:\n matchLabels:\n role: mongo\n environment: test\n serviceName: \"mongo\"\n replicas: 3\n ...\n ...\n volumeClaimTemplates:\n - metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: \"openebs-csi-sc\"\n accessModes: [\"ReadWriteOnce\"]\n resources:\n requests:\n storage: 15Gi\n\nNow delete the statefulset without deleting the statefulset pods for not to have any down time for application and apply the updated volumeClaimTemplate.\n\n system@master$ kubectl delete sts mongo -n mongo-ns --cascade=false\n statefulset.apps \"mongo\" deleted\n \n system@master$ kubectl apply -f mongo-sts.yaml \n statefulset.apps/mongo created\n service/mongo unchanged\n\nVerify changes by describing the sts\n\n system@master$ kubectl describe sts mongo -n mongo-ns\n \n Name: mongo\n Namespace: mongo-ns\n CreationTimestamp: Mon, 17 Aug 2020 16:38:51 +0530\n Selector: environment=test,role=mongo\n ...\n ...\n Volume Claims:\n Name: mongo-persistent-storage\n StorageClass: openebs-csi-sc\n Labels: \n Annotations: \n Capacity: 15Gi\n Access Modes: [ReadWriteOnce]\n\nIn the above output, capacity has been updated from **5Gi** to **15Gi**. So successfully updated the statefulset capacity without any down time.\n","notHasFeatureImage":false,"slug":"resize-kubernetes-statefulsets-without-impact"},{"id":26,"title":"Handling node down/cluster scaling on LocalPV backed workloads","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"21-08-2020","tags":["LocalPV","OpenEBS"],"excerpt":"In this article, we'll discuss the steps that is getting handled when a node down/cluster scaling on LocalPV backs workloads.","content":"\nKubernetes is increasingly used for running production-grade stateful services. Organizations are making progress on a containerized form of their production workloads for running in Kubernetes. There are already solutions available for the containerized version of stateful applications, network, storage, etc.\n\n### Handling node down / cluster scaling on Local PV backed workloads\n\nOpenEBS is one of the leading containerized storage solutions for Kubernetes, and it is a rapidly growing Sandbox project in CNCF. MayaData is the primary maintainer and contributor of OpenEBS along with other companies. MayaData also contributed another open source project, Litmus, into CNCF, which does mostly Chaos engineering in Kubernetes, which helps SREs and developers to do all kinds of testing of their applications and components in Kubernetes before going into production.\n\nIt is a must requirement of a persistent storage solution for running the stateful application, be it a **Deployment** or **StatefulSet**. OpenEBS provides many storage engines, and each storage engine is suitable for specific applications or workloads. Some engines provide storage level synchronous replication, capable of taking snapshots and cloning, backup and restore, volume expansion, CSI complaint, performance-oriented, etc. So choosing the engine based on the workload requirement is an important activity.\n\nOpenEBS provides dynamic provisioning of LocalPV using an external device, and this external device will be allocated entirely to an application. You can also use the partitioned disk for using OpenEBS LocalPV by using the `openebs-hostpath` storage engine. In this article, we provisioned a MySQL deployment on an OpenEBS LocalPV device dynamically.\n\nThis article is a step-by-step instruction. We will mention how a MySQL application deployment running on ***OpenEBS LocalPV*** device volume is getting handled when a Node down scenario or a cluster scale down situation happens in the GKE cluster. In GKE and some other managed clusters like EKS, the node name will change if the cluster undergoes a scale down and scale-up operation has performed. So the application running on the ***OpenEBS LocalPV*** device will not be able to attach to the new node since the corresponding PV has volume node affinity. We need to update the new node name details in PV, where the disk got attached.\n\nIn this article, we discuss the steps that need to be performed to make the application into a running state when a Node down / scale down cluster scenario has happened. This situation is usually required in case of managed clusters where the node name will get changed during this scenario. As stated earlier, the following approach works fine for both Deployment type and StatefulSet type, but ensure that the following steps are correctly satisfied. Let’s start with the step by step instructions once you have scaled up the cluster after a scale down scenario.\n\n1. Verify all nodes are now in Ready state.\n ```\n $ kubectl get node\n NAME STATUS ROLES AGE VERSION\n gke-openebs-mysql-default-pool-d55297a7-bjjp Ready 74s v1.16.13-gke.1\n gke-openebs-mysql-default-pool-d55297a7-j1vm Ready 80s v1.16.13-gke.1\n gke-openebs-mysql-default-pool-d55297a7-pvg4 Ready 85s v1.16.13-gke.1\n ```\n2. Ensure OpenEBS pods are in Running state.\n ```\n $ kubectl get pod -n openebs\n NAME READY STATUS RESTARTS AGE\n maya-apiserver-76cb4df9b8-wpbf6 1/1 Running 0 22m\n openebs-admission-server-5cf696b8d5-d97bn 1/1 Running 0 22m\n openebs-localpv-provisioner-7654f6dbd9-hskq8 1/1 Running 0 22m\n openebs-ndm-7dtts 1/1 Running 0 2m19s\n openebs-ndm-c4r4m 1/1 Running 0 2m23s\n openebs-ndm-lnb5c 1/1 Running 0 2m12s\n openebs-ndm-operator-6cfc59b69b-684nx 1/1 Running 0 22m\n openebs-provisioner-7d9884d4ff-tfcxj 1/1 Running 0 22m\n openebs-snapshot-operator-7ff577c889-kfttj 2/2 Running 0 22m\n ```\n3. Check the status of the application pod. It will be in the `Pending` state.\n ```\n $ kubectl get pod\n NAME READY STATUS RESTARTS AGE\n percona-9fbdb8678-lncd5 0/1 Pending 0 17m\n ```\n4. Label all the nodes with the same custom label used in the `nodeSelector` field in the STS app. In my case, there is no custom node label used in application deployment. So we are skipping this step.\n\n5. Attach the disk randomly to any node in the same zone. Note down the device name and node name where it is getting attached. This information will be needed in step 9.\n ```\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-bjjp --disk mysql-disk1 --device-name mysql-disk1 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-j1vm --disk mysql-disk2 --device-name mysql-disk2 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-pvg4 --disk mysql-disk3 --device-name mysql-disk3 --zone=us-central1-c\n ```\n6. Verify BDs are updated with new node names\n ```\n $ kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n blockdevice-4f51859193d333687a873af7acf8ad78 gke-openebs-mysql-default-pool-d55297a7-j1vm 32212254720 Unclaimed Active 37m\n blockdevice-967d7816c2a2d73b91c8c6310dc70465 gke-openebs-mysql-default-pool-d55297a7-bjjp 32212254720 Claimed Active 37m\n blockdevice-ddfc782ea661fc9007a896438f483e3d gke-openebs-mysql-default-pool-d55297a7-pvg4 32212254720 Unclaimed Active 37m\n ```\n7. Get the PV details of the associated application\n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 5G RWO Delete Bound default/demo-vol1-claim openebs-device 33m\n ```\n8. Create a directory and copy the YAML spec of all the associated PVs into it like below\n ```\n $ mkdir mysql-restore\n $ cd mysql-restore/\n $ kubectl get pv pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 -o yaml --export > pv1.yaml\n ```\n Note: If it is StatefulSet, take the YAML spec of all the associated PVs of that application.\n\n9. Modify the above-copied YAML with the new hostname in the copied YAML of PV. The following is that snippet of PV spec where it mentions the new node name where the Local disk is attached.\n ```\n path: /dev/disk/by-id/scsi-0Google_PersistentDisk_mysql-disk1\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - gke-openebs-mysql-default-pool-d55297a7-bjjp\n ```\n10. Now get the PV and then delete the PV\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 5G RWO Delete Bound default/demo-vol1-claim openebs-device 36m\n \n $ kubectl delete pv pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999\n persistentvolume \"pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999\" deleted\n ```\n The deletion of the PV will not be completed since it has the finaliser set with the PV. So we need to cancel the ongoing operation and then edit the PV and remove Finalizers. Once finalizers are removed, the volume will be automatically deleted.\n\n11. Verify that the PV of the application has been removed successfully.\n ```\n $ kubectl get pv\n No resources were found in the default namespace.\n ```\n12. Now, apply the updated YAML files of the PV.\n ```\n $ kubectl apply -f pv1.yaml \n ```\n Note: Perform the same for other PVs as well if the application is a StatefulSet.\n\n13. Verify that if PODs are started `Running` from `Pending` state.\n ```\n $ kubectl get pod -o wide\n NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\n percona-9fbdb8678-lncd5 1/1 Running 0 29m 10.16.0.2 gke-openebs-mysql-default-pool-d55297a7-bjjp \n ```\n14. Log in to the application and verify that you are able to access the data.\n\n ```\n $ kubectl exec -it percona-9fbdb8678-lncd5 sh\n kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.\n sh-4.2$ mysql -uroot -pk8sDem0;\n mysql: [Warning] Using a password on the command line interface can be insecure.\n Welcome to the MySQL monitor. Commands end with ; or \\g.\n Your MySQL connection id is 2\n Server version: 5.7.30-33 Percona Server (GPL), Release 33, Revision 6517692\n \n Copyright (c) 2009-2020 Percona LLC and/or its affiliates\n Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.\n \n Oracle is a registered trademark of Oracle Corporation and/or its\n affiliates. Other names may be trademarks of their respective\n owners.\n \n Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\n mysql> SHOW DATABASES;\n +--------------------+\n | Database |\n +--------------------+\n | information_schema |\n | mysql |\n | performance_schema |\n | pets |\n | sys |\n +--------------------+\n 5 rows in set (0.07 sec)\n \n mysql> use pets;\n \n Reading table information for completion of table and column names\n You can turn off this feature to get a quicker startup with -A\n \n Database changed\n mysql> SELECT * FROM cats;\n +----+---------+--------+------------+\n | id | name | owner | birth |\n +----+---------+--------+------------+\n | 1 | Sandy | Lennon | 2015-01-03 |\n | 2 | Cookie | Casey | 2013-11-13 |\n | 3 | Charlie | River | 2016-05-21 |\n +----+---------+--------+------------+\n 3 rows in set (0.00 sec)\n ```","notHasFeatureImage":false,"slug":"handling-node-down/cluster-scaling-on-localpv-backed-workloads"},{"id":27,"title":"OpenEBS-Velero backup/restore of stateful applications","author":"Vishnu Itta","author_info":"Developer who is always eager to learn, loves math, algorithms, & programming. Have a good experience in storage protocols, ZFS, FreeBSD internals, Linux, device drivers.","date":"04-08-2020","tags":["OpenEBS","Velero"],"excerpt":"In this blog, we will go through the backup/restore solution provided to its users by OpenEBS with the help of Velero.","content":"\n### **Backup/Restore of stateful applications in OpenEBS**\n\nSREs play a crucial role in automating operations. This role includes handling infrastructure upgrades and the upgrades of software running on that infrastructure. When running stateful workloads, the data must be backed up before any upgrades.\n\nAdditionally, workloads on Kubernetes are often different than traditional monolithic applications since Kubernetes supports microservices and loosely coupled workloads. This brings-in the need for a `cloud-native` design of backup/restore per workload or application. Once implemented this approach provides benefits to users such as an app-centric view, ease of management, setting of RPO/RTO at the workload level, and others. To learn more about cloud native backup drivers and requirements - please read this [thenewstack.io article](https://thenewstack.io/cloud-native-backups-disaster-recovery-and-migrations-on-kubernetes/).\n\nThe below graphic shows microservices, and their loosely coupled nature. Managing backups and the tuning of RPO/RTO per microservice is required to preserve the loosely coupled nature of these workloads and of the teams responsible for them.\n![OpenEBS Velero backup/restore of stateful applications: fig 1](https://lh4.googleusercontent.com/JyIxZZtj-1rBOGo2z1bKcYlD9-halM8dXpChtOIPro_aZEhQObTL_K5_Be_hLgqXl7aT68jYqFjNE9C6rZK0IRaV_neH4DURZhUr3z5FkVirzMirk_z8FiscY6_sb3JPhlAx1XRL)\nThe following are **challenges** in redesigning backup/restore of K8s applications:\n\n- Ability to look from the application view rather than only the data\n- Dependencies with K8s constructs like node names, PVCs, storage classes, topologies in them\n- Snapshots of selected volumes rather than entire mount point/disk in hyper-converged infrastructure\n- K8s application deployment workflow to copy data during a restore whether using operators or another method\n\nIn this blog, we will go through a backup/restore solution for ***OpenEBS*** with the help of ***Velero***. OpenEBS is leading open source container attached storage solution, and Velero is one of the most used open-source tools for backups in Kubernetes environments.\n\n### **Backup/Restore solution**\n\nThis solution, with Velero, provides a declarative way for users to specify\n\n- object storage as the destination location to store backed-up data\n- K8s applications and its resources which are part of the backup\n- storage provider configuration to backup/restore data\n- schedules to take a backup at regular intervals\n- restore selected resources from the selected backup\n- app-specific hooks to execute before and after performing backup/restore\n\nLet's start by examining what OpenEBS, as container attached storage, offers. It has many kinds of storage engines as below:\n\n- Local PV host path and block device\n- Jiva\n- CStor\n- Local PV ZFS\n\n### **Local PV / Jiva Volumes + Restic + Restore Item Action plugin**\n\nOpenEBS Local PV volumes are prominent among cloud-native applications. These applications themselves take care of replicating data and snapshotting. As the name suggests, these are used in hyper-converged infrastructure. This brings in a few challenges during backup/restoration.\n\nLocal PVs related storage classes will have `WaitForFirstConsumer`. It provides delayed binding of volumes until Pod gets scheduled on the node. K8s sets the node name as an annotation on PVC. Restoring that PVC onto the destination cluster will leave the pod in pending state. This solution takes care of it by applying node name mapping in PVC annotation. Restore Item Action plugin, contributed by OpenEBS, in Velero performs this mapping.\n\nVelero's in-built file-based `Restic` plugin helps in backing up the application's data. `Init` container will be injected into the application to copy data from the destination location to the volume.\n\nJiva volumes are used by applications that need replication functionality from storage providers. File-based `Restic` plugin can be used to backup/restore applications using jiva volumes.\n\n### **CStor / Local PV ZFS + Velero plugin**\n\nStandard applications use OpenEBS cStor volumes. These applications need replication, snapshotting, cloning, etc., from storage providers. Also, these applications are free to run-anywhere-in-the-cluster nature. OpenEBS have a cStor velero-plugin for crash-consistent backup of data. This plugin performs pausing of IOs on the volume before the snapshot is taken on all the replicas. It resumes IOs once the point-in-time snapshot is taken. It reads the snapshot content from one of the replicas and backs it up at the object location. When a Schedule is newly created, full data will be backed up at the destination location. The plugin manages the completion status of the backup. On the next iterations, the plugin just backups the changes from previous successful backup.\n\nBased on the replication settings of cStor at the destination, the plugin takes care of copying data to the configured number of replicas. The connection between iSCSI target pod and replica pods is not established until data is copied into the volume. This avoids usage of the PV by any application during restore time.\n\nCStor plugin also provides an option to keep backups locally with the main volume in the form of local snapshots. Users can create a backup of applications and with local snapshots. Restoring these backups, which consists of local snapshots, can be done in a different namespace of the same cluster.\n\nWork-related to local PV ZFS plugin is in-active development to provide all the features as mentioned above and much more.\n\n### **Summary**\n\nTo summarize, a new kind of thinking (or) a new kind of approach is required to do cloud-native backups. OpenEBS has made tremendous progress in achieving it and provides various features and flavors of backup/restoration of its volumes.\n\n- File-based\n- Point-in-time block-based local snapshot and restore\n- Full and incremental block-based remote backup and restore\n- Per-workload backups\n- K8s resources transformations\n\nCredits to the Velero team for building a wonderful open-source backup/restore tool.\n\nPlease visit [https://openebs.io/docs](/docs), [https://velero.io](https://velero.io/) for more details about storage class, backup/restore examples, Velero.\n","notHasFeatureImage":false,"slug":"openebs-velero-backup/restore-of-stateful-applications"},{"id":28,"title":"Kubernetes StatefulSet on ppc64le using OpenEBS LocalPV provisioner","author":"Peeyush Gupta","author_info":"Peeyush, Sr. Developer Advocate, DigitalOcean, is a cloud enthusiast with 5+ years of experience in developing cloud platforms and helping customers migrate their legacy applications to the cloud.","date":"16-07-2020","tags":["OpenEBS","Kubernetes","LocalPV"],"excerpt":"In this blog, we'll explain how to install OpenEBS on Kubernetes StatefulSet running on the ppc64le platform & to using the OpenEBS LocalPV provisioner to deploy a StatefulSet.","content":"\nGuest post by Peeyush Gupta, Sr. Developer Advocate, DigitalOcean\n\n**OpenEBS** is the leading open-source project for container-attached and container-native storage on **Kubernetes**. OpenEBS adopts Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller. OpenEBS implements granular storage policies and isolation that enable users to optimize storage for each specific workload. OpenEBS runs in userspace and does not have any Linux kernel module dependencies. Here is how the setup looks like for OpenEBS LocalPV hostpath:\n![Kubernetes StatefulSet on ppc64le using OpenEBS LocalPV provisioner](https://lh4.googleusercontent.com/-erccwTcJCmyJGswEZ3Pul1-pvJO-kvn34nr22mqYumR1IHUVhX8BWOeennt1u91EYUKtpUfAPBSiP1XD_1z6XYmG8Tlywvl9GellLpkr8EyYTFLXT3YpIZ_nneRcen_G8uKVV6Q)\nIn this tutorial, we will see how we can install OpenEBS on Kubernetes running on the ppc64le platform. Then we will use the OpenEBS LocalPV provisioner to deploy a StatefulSet. The ppc64le servers used in this tutorial are running on IBM Cloud.\nKubernetes support multi-arch platforms. You can deploy the cluster on ppc64le based servers using kubeadm. For this tutorial, I am using a 3 node cluster:\n\n root@openebs-k8s-server:~# kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n openebs-k8s-server Ready master 17h v1.18.5\n openebs-worker-1 Ready 17h v1.18.5\n openebs-worker-2 Ready 17h v1.18.5\n\nWe will be deploying OpenEBS lite on this cluster. Here is an excellent blog post on how to do that: [https://help.mayadata.io/hc/en-us/articles/360031969532-Installing-OpenEBS-with-only-Local-PV-support](https://help.mayadata.io/hc/en-us/articles/360031969532-Installing-OpenEBS-with-only-Local-PV-support). To get started, deploy the openebs-operator-lite.yaml file for ppc64le using:\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-lite-ppc64le.yaml\n\nOnce deployed, you should be able to see the node-disk-manager and LocalPV components running. Note that node-disk-manager is not a mandatory component for provisioning LocalPV hostpath volumes. OpenEBS LocalPV provisioner can be installed and run standalone as well.\n\n root@openebs-k8s-server:~# kubectl get pods --all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n kube-system coredns-66bff467f8-njjc9 1/1 Running 0 17h\n kube-system coredns-66bff467f8-tndsx 1/1 Running 0 17h\n kube-system etcd-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-apiserver-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-controller-manager-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-proxy-55fbj 1/1 Running 0 17h\n kube-system kube-proxy-gt5rw 1/1 Running 0 17h\n kube-system kube-proxy-l5pz2 1/1 Running 0 17h\n kube-system kube-scheduler-openebs-k8s-server 1/1 Running 0 17h\n kube-system weave-net-c2gmk 2/2 Running 1 17h\n kube-system weave-net-qp5c7 2/2 Running 0 17h\n kube-system weave-net-trgr6 2/2 Running 1 17h\n openebs openebs-localpv-provisioner-7bb7dc9958-ln284 1/1 Running 0 16h\n openebs openebs-ndm-kqhpr 1/1 Running 0 16h\n openebs openebs-ndm-nxswk 1/1 Running 0 16h\n openebs openebs-ndm-operator-746d6cd4fd-bm2fp 1/1 Running 1 16h\n\nNext, we will deploy the storage class. This storage class will be used with PersistentVolumeClaim to create volumes.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-lite-sc.yaml \n\nThis will create 2 storage classes, openebs-device, and openebs-hostpath, on the cluster. \n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get sc\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\n openebs-device openebs.io/local Delete WaitForFirstConsumer false 6m12s\n openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 6m12s\n\nIn our case, we are interested in the `openebs-hostpath` storage class for this tutorial. Now, we are ready to deploy the StatefulSet that will consume the volume created using the above storage class. Here is a sample StatefulSet:\n\n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: local-test\n spec:\n serviceName: \"local-service\"\n replicas: 1\n selector:\n matchLabels:\n app: local-test\n template:\n metadata:\n labels:\n app: local-test\n spec:\n containers:\n - name: test-container\n image: busybox\n command:\n - \"/bin/sh\"\n args:\n - \"-c\"\n - \"sleep 100000\"\n volumeMounts:\n - name: openebs-localpv-hostpath\n mountPath: /usr/test-pod\n volumeClaimTemplates:\n - metadata:\n name: openebs-localpv-hostpath\n spec:\n accessModes: [ \"ReadWriteOnce\" ]\n storageClassName: \"openebs-hostpath\"\n resources:\n requests:\n storage: 5G\n\nThe above YAML creates a StatefulSet named `local-test`, which has a container named `test-container`. This container has a volume mounted at path `/usr/test-pod`. The claim for this particular volume references the `openebs-hostpath` storage class. We will save this YAML using the name openebs-localpv-stateful.yaml, and this can be deployed using kubectl create:\n\n kubectl apply -f openebs-localpv-stateful.yaml\n\nYou can verify the respective pv and pvc:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n openebs-localpv-hostpath-local-test-0 Bound pvc-e61a4156-b0bb-4199-b991-9c42b1830ec5 5G RWO openebs-hostpath 17s\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e61a4156-b0bb-4199-b991-9c42b1830ec5 5G RWO Delete Bound default/openebs-localpv-hostpath-local-test-0 openebs-hostpath 18s\n\nOnce the volume is bound to the claim, the relevant pod i.e. local-test-0 comes up in running state:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pods --all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n default local-test-0 1/1 Running 0 13s\n kube-system coredns-66bff467f8-njjc9 1/1 Running 0 21h\n kube-system coredns-66bff467f8-tndsx 1/1 Running 0 21h\n kube-system etcd-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-apiserver-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-controller-manager-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-proxy-55fbj 1/1 Running 0 21h\n kube-system kube-proxy-gt5rw 1/1 Running 0 21h\n kube-system kube-proxy-l5pz2 1/1 Running 0 21h\n kube-system kube-scheduler-openebs-k8s-server 1/1 Running 0 21h\n kube-system weave-net-c2gmk 2/2 Running 1 21h\n kube-system weave-net-qp5c7 2/2 Running 0 21h\n kube-system weave-net-trgr6 2/2 Running 1 21h\n openebs openebs-localpv-provisioner-6fc9664557-njvbg 1/1 Running 0 24m\n openebs openebs-ndm-754qs 1/1 Running 0 24m\n openebs openebs-ndm-operator-798f74c6b9-24jvv 1/1 Running 0 24m\n openebs openebs-ndm-x96ks 1/1 Running 0 24m\n\nIf you describe the pvc, you can see the volume is being provisioned using openebs-localpv-provsioner:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl describe pvc\n Name: openebs-localpv-hostpath-local-test-0\n Namespace: default\n StorageClass: openebs-hostpath\n Status: Bound\n Volume: pvc-b89a1c70-beeb-4bdc-a555-348b01832443\n Labels: app=local-test\n Annotations: pv.kubernetes.io/bind-completed: yes\n pv.kubernetes.io/bound-by-controller: yes\n volume.beta.kubernetes.io/storage-provisioner: openebs.io/local\n volume.kubernetes.io/selected-node: openebs-worker-1\n Finalizers: [kubernetes.io/pvc-protection]\n Capacity: 5G\n Access Modes: RWO\n VolumeMode: Filesystem\n Mounted By: local-test-0\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal WaitForFirstConsumer 13s persistentvolume-controller waiting for first consumer to be created before binding\n Normal ExternalProvisioning 13s (x2 over 13s) persistentvolume-controller waiting for a volume to be created, either by external provisioner \"openebs.io/local\" or manually created by system administrator\n Normal Provisioning 13s openebs.io/local_openebs-localpv-provisioner-6fc9664557-njvbg_884920b8-e474-4635-9daf-8b4a8f113b10 External provisioner is provisioning volume for claim \"default/openebs-localpv-hostpath-local-test-0\"\n Normal ProvisioningSucceeded 11s openebs.io/local_openebs-localpv-provisioner-6fc9664557-njvbg_884920b8-e474-4635-9daf-8b4a8f113b10 Successfully provisioned volume pvc-b89a1c70-beeb-4bdc-a555-348b01832443\n\nYou can cleanup the whole setup using:\n\n kubectl delete -f openebs-localpv-stateful.yaml\n kubectl delete -f openebs-lite-sc.yaml\n kubectl delete -f openebs-operator-lite-ppc64le.yaml\n\n### About the author:\n\nPeeyush Gupta is a cloud enthusiast with 5+ years of experience in developing cloud platforms and helping customers migrate their legacy applications to the cloud. He has also been a speaker at multiple meetups and loves to contribute to open-source projects. He is currently working with DigitalOcean as Sr. Developer Advocate.\n","notHasFeatureImage":false,"slug":"kubernetes-statefulset-on-ppc64le-using-openebs-localpv-provisioner"},{"id":29,"title":"Data Migration Within Kubernetes Clusters","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"02-07-2020","tags":["Kubernetes"],"excerpt":"In this blog, we'll talk about migrating data within Kubernetes from one node to another without any downtime of the application.","content":"\nIn large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. MayaData, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. cStor is one of the storage engines of OpenEBS. \nThis blog is for OpenEBS users, specifically **cStor CSI** users looking to **migrate data within Kubernetes** from one node to another without any downtime of the application.\n\n![Data Migration within Kubernetes from Node2 to Node3](/images/blog/Data-migration-diagram.png)\n\n### Create cStor Pools(CSPC):\n\nCreate cStor pools by following the steps mentioned [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md). Once the pools are created, wait till all the cStor pools marked as Healthy. Check the cStor pools status by executing `kubectl get cspc -n openebs` command(cspc - cStorPoolCluster)\n\n NAME HEALTHYINSTANCES PROVISIONEDINSTANCES DESIREDINSTANCES AGE\n cspc-stripe-pool 3 3 3 4m13s\n\nFollowing command `kubectl get cspi -n openebs`(cStorPoolInstances) will be helpful to know more information about the pool.\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY READONLY TYPE STATUS AGE\n cspc-stripe-pool-6qkw e2e1-node1 230k 9630M 10G false stripe ONLINE 21m\n cspc-stripe-pool-pn9p e2e1-node2 230k 9630M 10G false stripe ONLINE 12s\n cspc-stripe-pool-psz5 e2e1-node3 230k 9630M 10G false stripe ONLINE 21m\n\n### Create CSI Volume:\n\nCreate CSI volumes on cStor pools created above by following the steps mentioned [here](https://github.com/openebs/cstor-csi#provision-a-cstor-volume-using-openebs-cstor-csi-driver). As part of volume provisioning, a resource called `CStorVolumeConfig` will be created. Once the volume is provisioned successfully, then CVC(cStorVolumeConfig) status will update to Bound, which means all the CStorVolume resources are successfully created. Following is the command which will help to get CVC status `kubectl get cvc -n openebs`.\n\n NAME STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306 Bound 5m56s\n \n\n### **Inspect CVC**:\n\nInterfere CVC to know on which node data exists. When we do `kubectl get cvc -n -o yaml`.\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorVolumeConfig\n name: pvc-d1b26676-5035-4e5b-b564-68869b023306\n …\n …\n spec:\n capacity:\n storage: 5Gi\n ...\n ...\n policy:\n replicaPoolInfo:\n - poolName: cspc-stripe-pool-6qkw\n - poolName: cspc-stripe-pool-pn9p\n status:\n phase: Bound\n poolInfo:\n - cspc-stripe-pool-6qkw\n - cspc-stripe-pool-pn9p\n\nFrom the above info CStorVolumeReplicas(CVR) are scheduled on cStor pools **cspc-stripe-pool-6qkw** and **cspc-stripe-pool-pn9p** from the **status.poolInfo** since above pools are on **e2e1-node1** and **e2e1-node2** (able to find node info from cspi output) sodata also will be available on the same node. Info under spec i.e **spec.policy.replicaPoolInfo** will convey where to schedule cStorVolumeReplicas.\n\nTo know more details of CVR we can get from `kubectl get cvr -n openebs`\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.47G 1.26G Healthy 15h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.47G 1.26G Healthy 15h\n\n**Steps to migrate data from one node to other:**\n\n1. Scale the CStorVolumeReplica to the desired Node.\n2. Scale down the CStorVolumeReplicas from unwanted Node.\n\n**Step1: Scale the CStorVolumeReplicas to the desired node**\n\nGet cStor pool name, which doesn’t have corresponding volume CVR in it and add it under **spec.policy.replicaPoolInfo** of CVC.\n\nWe can get pool name name which doesn’t have CVR in it by comparing outputs of `kubectl get cspi -n openebs -l openebs.io/cstor-pool-cluster=` and `kubectl get cvc -o yaml` as stated inspect CVC. In this example CVR of volume pvc-d1b26676-5035-4e5b-b564-68869b023306 doesn’t not exist in cStor pool **cspc-stripe-pool-psz5.** After finding the pool name add it under **policy.replicaPoolInfo** list in CVC.\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorVolumeConfig\n name: pvc-d1b26676-5035-4e5b-b564-68869b023306\n …\n …\n spec:\n …\n ...\n policy:\n replicaPoolInfo:\n - poolName: cspc-stripe-pool-6qkw\n - poolName: cspc-stripe-pool-pn9p\n - poolName: cspc-stripe-pool-psz5\n …\n …\n status:\n poolInfo:\n - cspc-stripe-pool-6qkw\n - cspc-stripe-pool-pn9p\n - cspc-stripe-pool-psz5\n\n**Superb!**\nOnce the pool was added into the `spec.replicaPoolInfo` then the status of CVC will be updated with a new pool name as shown above, and raise an event which means that CVR was created on a newly added pool. We can get the CVR status by executing `kubectl get cvr -n openebs`\n\n**Events**: Events on corresponding CVC\n\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal ScalingVolumeReplicas 14s (x2 over 15h) cstorvolumeclaim-controller successfully scaled volume replicas to 3\n\n**CVR status(by executing the command)**:\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.48G 1.26G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-psz5 91.4M 42.4M ReconstructingNewReplica 33s\n\nIn the above output, newly created CVRs convey that it reconstructed data from scratch by talking to peer replicas. Wait till the newly created CVR marked as **Healthy**. To know status periodically execute `kubectl get cvr -n openebs` command.\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-psz5 1.48G 1.25G Healthy 5m28s\n\nNote:\n\n- Reconstructing will take time, depending on the amount of data it has to rebuild.\n\n**Step2: Scale down the CStorVolumeReplicas from unwanted nodes**\n\nOnce the newly created CVR is marked as Healthy, then we can remove the unwanted pool name from Spec of CVC replicaInfo and save it. \n\nIn this example, I need to remove the data from the pool **cspc-stripe-pool-pn9p**, scheduled on **e2e1-node2**. Once the pool name is removed from CVC **spec.policy.replicaPoolInfo**, then corresponding CVR in that pool will be deleted. CVC will generate events and status of CVC also will be updated.\n\nEvents on CVR:\n\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Warning ScalingVolumeReplicas 4s (x2 over 64m) cstorvolumeclaim-controller Scaling down volume replicas from 3 to 2 is in progress\n Normal ScalingVolumeReplicas 4s (x2 over 64m) cstorvolumeclaim-controller successfully scaled volume replicas to 2\n\nFrom output of `kubectl get cspi -n openebs`\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY READONLY TYPE STATUS AGE\n cspc-stripe-pool-6qkw e2e1-node1 1260M 8370M 9630M false stripe ONLINE 17h\n cspc-stripe-pool-pn9p e2e1-node2 230k 9630M 9630M false stripe ONLINE 16h\n cspc-stripe-pool-psz5 e2e1-node3 1260M 8370M 9630M false stripe ONLINE 17h\n\n**Perfect!**\n\nFrom the above storage usage, I can successfully migrate the data from one node to another without any downtime of the application.\n","notHasFeatureImage":false,"slug":"data-migration-within-kubernetes-clusters"},{"id":30,"title":"Restricting cStor pool usage within a specified threshold value","author":"Giridhara Prasad","author_info":"Lead Engineer at Mayadata Inc. Giridhar is experienced in software test automation, chaos engineering. Currently, he's working on Litmus, an Open Source chaos engineering project.","date":"20-05-2020","tags":["OpenEBS","cStor"],"excerpt":"Learn how to restrict cStor pool usage within a specified threshold value","content":"\ncStor is one of the storage engines provided by OpenEBS. The integral component of the cStor engine is its storage pool from which the volumes are created. The storage pool is constructed with the collection of block devices. When the pool is completely utilized, it may misbehave in such a way that the pool itself cannot be imported successfully to recover from failures.\n\nIn order to overcome this situation, cStor recommends the optimal usage of storage capacity in the pool by restricting the write I/Os by converting it into read-only when the threshold limit is exceeded. While creating cStor SPC, the field *roThresholdLimit* has to be specified in percentage value under pool spec as follows:\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-pool\n type: disk\n poolSpec:\n poolType: striped\n roThresholdLimit: 80\n blockDevices:\n blockDeviceList:\n - blockdevice-31e0768585cb80ed2352affa73ec94e2\n - blockdevice-ab636ddeba8f8cd45f7e91a6b55c15e5\n - blockdevice-75275112e966e43c2ac1311a7a492fac\n\nIn the above snippet, *roThresholdLimit: 80*  indicates that the pool will become read-only when the usage exceeds 80% of its total capacity. Upon trying to create the above SPC, the following CSPs will be created.\n\n NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE AGE\n cstor-pool-0vl0 45G 4.7G 49.8G Healthy true striped 2m8s\n cstor-pool-qnm1 77K 49.7G 49.8G Healthy false striped 2m8s\n cstor-pool-x4gj 77K 49.7G 49.8G Healthy false striped 2m8s\n\nThe read-only status of each CSP is indicated, as shown in the above template.\n\nWhen the usage of the pool crosses 80% of its actual capacity, it will become read-only. It means all the replicas in that pool won’t serve further write IOs irrespective of the amount of space they consumed. As an impact, if the persistent volumes do not have enough healthy storage replicas, then the volume will become read-only.\n\nThe status of active replica where write IOs were happening turns offline when the pool becomes read-only whereas, for the idle replica, it remains healthy. The sample output of CVR is as below. Here, one replica is in Offline state as its pool is in a read-only state.\n\n k8s@master:kubectl get cvr -n openebs\n NAME USED ALLOCATED STATUS AGE\n pvc-261d6832-8b23-476c-8aa3-b95104e20030-cstor-pool-0vl0 1.15G 1.04G Offline 23m\n pvc-f38f5517-a7bc-492d-a6eb-27ac510ced3b-cstor-pool-qnm1 74.7K 17.7K Healthy 23m56s\n\nThe *roThresholdLimit* is the configurable value. In case, if you want to increase/decrease the percentage value, it has to be modified at each CSP level. Editing at SPC config won’t be effective. In case if the administrator didn’t set the *roThresholdLimit* field during SPC creation, the default value would be set to, 85% considering it as the optimal value for usage. Administrators can set the roThresholdLimit value in CSP from 0 to 100 though the OpenEBS team won't recommend setting it to 100 percent.\nWhen the pool became read-only, the administrator can either increase the pool capacity by executing the steps specified [here](https://github.com/openebs/openebs-docs/blob/day_2_ops/docs/cstor_add_disks_to_spc.md) or increase the roThresholdLimit value in that CSP to make pool RW.\n","notHasFeatureImage":false,"slug":"restricting-cstor-pool-usage-within-a-specified-threshold-value"},{"id":31,"title":"Getting started with K3s in vSphere and OpenEBS cStor","author":"Giridhara Prasad","author_info":"Lead Engineer at Mayadata Inc. Giridhar is experienced in software test automation, chaos engineering. Currently, he's working on Litmus, an Open Source chaos engineering project.","date":"26-03-2020","tags":["OpenEBS"],"excerpt":"In this blog, more of a tutorial, I will walk you through the steps to install K3OS and setup OpenEBS.","content":"\n[K3OS](https://github.com/rancher/k3os/) is a Linux distribution built to run lightweight Kubernetes clusters called [K3s](https://github.com/rancher/k3s/). It is specifically designed only to have what is needed to run [k3s](https://github.com/rancher/k3s).\n\nIn this blog, more of a tutorial, I will walk you through the steps to install K3OS and setup OpenEBS, a CNCF project, and leading Open Source Container Attached Storage solution for Kubernetes Stateful Workloads.\n\n#### **Setting up K3OS in vSphere**\n\nK3OS kernel is forked from Ubuntu-18.04 LTS, and its userspace binaries are from alpine. So, you need to select Ubuntu Linux (64 bit) as the guest operating system while creating a virtual machine.\n\n![Select guest operating system](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage.png)\n\nDownload the latest K3OS iso file (currently v0.9.0) from its [GitHub release](https://github.com/rancher/k3os/releases) page. Attach the iso file into a virtual machine and start it with the live installation option, as shown below.\n\nSelect the option *K3OS LiveCD & install* and boot the operating system.\n\n![Live installation](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-3.png)\n\nAfter booting up successfully, you will be landed in a login prompt. The default user in K3OS is rancher. You can login as rancher user without any password.\n\n![Login prompt](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-6.png)\n\nAfter performing a live install, You need to install the Operating System into a disk and can configure the machine either as a server(Master) or an agent(worker). This can be performed by executing the command `sudo k3os install`.\n\nSelect option 1. Install to disk to install K3OS into the disk. In the preceding questions, set up a new password for rancher user for enabling ssh communication to the server.\n\n![Installing into disk](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-4.png)\n\n### \n**Installing into disk**\n\nYou need to select either server or agent to install the relevant components in the machine. Select 1.server to deploy K3s server components. You can set up a token or cluster secret that could be used while joining K3s agents to the server.\n\n![server installation](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-7.png)\n\nAfter completing the installation, a screen similar to the following one will be displayed.\n\n![Login prompt](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-1.png)\n\nThus, the K3s server can be configured successfully. In case if DHCP is not configured, you need to assign an IP address and other networking details using connmanctl utility. Login into the server as rancher user and enter the password configured in the previous step.\n\nLet us find the connman network service bound to the eth0 device by executing the below command.\n\n sudo connmanctl services\n\nThe above command will list the services below.\n\n![connmanctl services](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-5.png)\n\nAfter identifying the service, you can assign the IP address, netmask, gateway, and DNS server through the following command.\n\n sudo connmanctl config --ipv4 manual --nameservers \n\nAfter executing the above command, ensure if the network is configured correctly through `ifconfig` command.\n\nReboot the machine after setting up networking.\n\n#### **Install K3s agent**\n\nIn K3s nomenclature, Kubernetes workers are called as agents. While installing k3os into a disk, you need to select the option 2. agent to configure K3s agent in the machine.\n\n![Run K3s agent](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-2.png)\n\nAfter selecting Agent, you need to provide the URL of the server to which the agent has to be configured. The URL of the k3s server could be formed in the following way.\n\n https://:6443\n\nAfter entering the URL, you need to provide the cluster secret, which was configured during server installation.\n\nAfter providing all the above inputs, initiate the agent deployment.\n\nConfigure networking in the same way as performed above for server and reboot the machine.\n\nAfter rebooting the agent machine, check the cluster status in the server as follows.\n\n k3os-1374 [~]$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n k3os-1374 Ready master 10m v1.17.2+k3s1\n k3os-15360 Ready 10m v1.17.2+k3s1\n k3os-1091 Ready 10m v1.17.2+k3s1\n\nCheck if all the cluster components are configured successfully and all the pods are running successfully by executing the below command.\n\n k3os-1374 [~]$ kubectl get pods -n kube-system\n NAME READY STATUS RESTARTS AGE\n helm-install-traefik-nmjvj 0/1 Completed 0 3d\n svclb-traefik-gp9ff 2/2 Running 1 2d23h\n svclb-traefik-qgdlx 2/2 Running 0 2d23h\n local-path-provisioner-58fb86bdfd-wkdtm 1/1 Running 1 3d\n metrics-server-6d684c7b5-mrxsr 1/1 Running 0 3d\n svclb-traefik-c4v7l 2/2 Running 0 3d\n coredns-d798c9dd-td5tr 1/1 Running 0 3d\n traefik-6787cddb4b-n57jz 1/1 Running 0 3d\n\n#### **Install OpenEBS**\n\nOpenEBS is a CNCF project delivering persistent block storage to the workloads deployed in Kubernetes. [cStor](/docs/concepts/cstor?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807) is one of the storage engines provided by OpenEBS besides [Jiva](/docs/concepts/jiva?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807) and [Local PV](/docs/concepts/localpv?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807).\n\ncStor was not supported in K3OS till k3os-v0.8.0 due to this [issue](https://github.com/rancher/k3os/issues/151). This issue has been addressed in v0.9.0 by adding udev support.\n\n k3os-1374 [~]$ kubectl apply -f openebs-operator-1.7.0.yaml \n namespace/openebs created\n serviceaccount/openebs-maya-operator created\n clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created\n clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created\n deployment.apps/maya-apiserver created\n service/maya-apiserver-service created\n deployment.apps/openebs-provisioner created\n deployment.apps/openebs-snapshot-operator created\n configmap/openebs-ndm-config created\n daemonset.apps/openebs-ndm created\n deployment.apps/openebs-ndm-operator created\n deployment.apps/openebs-admission-server created\n deployment.apps/openebs-localpv-provisioner created\n\nCheck if all the OpenEBS components are running successfully.\n\n k3os-1374 [~]$ kubectl get pods -n openebs\n NAME READY STATUS RESTARTS AGE\n openebs-admission-server-f67f77588-8kl78 1/1 Running 0 65s\n openebs-provisioner-7b8c68bf44-7bjw8 1/1 Running 0 66s\n openebs-ndm-qp26v 1/1 Running 0 66s\n openebs-ndm-84zb4 1/1 Running 0 66s\n openebs-ndm-dzghs 1/1 Running 0 66s\n openebs-localpv-provisioner-5c87bbd974-55486 1/1 Running 0 65s\n openebs-ndm-operator-5fccfb7976-dvhj6 1/1 Running 0 66s\n openebs-snapshot-operator-6c4c64d4bc-qxdwd 2/2 Running 0 66s\n maya-apiserver-84785d7fbd-ck7sh 1/1 Running 0 66s\n\nOpenEBS cStor engine requires external disks to be attached to the agents which group to form cStor Pools.\n\nThe disks or block devices are managed by the component called *Node disk manager*, shortly called as *NDM*. After attaching the disks to agent machines, check the block devices by executing the following command.\n\n k3os-1374 [~]$ kubectl get blockdevices -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n blockdevice-30a3eb18f5b9e2d470de45e39f1036b0 k3os-15360 17179869184 Unclaimed Active 1h\n blockdevice-86fc964305abe8970fc1508538a61dbc k3os-1374 17179869184 Unclaimed Active 1h\n blockdevice-b8735721689d5843bca10e7028f60a4e k3os-1091 17179869184 Unclaimed Active 1h\n\nIn this case, one block device has been attached to each K3s agent machine. Let us populate these block devices in the below pool creation manifest under *spec.blockDevices* and create the pool.\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-disk-pool\n type: disk\n poolSpec:\n poolType: striped\n blockDevices:\n blockDeviceList:\n - blockdevice-30a3eb18f5b9e2d470de45e39f1036b0\n - blockdevice-86fc964305abe8970fc1508538a61dbc \n - blockdevice-b8735721689d5843bca10e7028f60a4e\n\nAfter applying the above definition, check if the pools are created successfully by executing the following command.\n\n k3os-1374 [~]$ kubectl get csp\n NAME ALLOCATED FREE CAPACITY STATUS TYPE AGE\n cstor-disk-pool-rgy3 270K 15.9G 15.9G Healthy striped 2m3s\n cstor-disk-pool-gij4 270K 15.9G 15.9G Healthy striped 2m2s\n cstor-disk-pool-09l1 270K 15.9G 15.9G Healthy striped 2m2s\n\nAll the pool instances should be healthy and each instance runs a pod which can be found by executing the following command.\n\n k3os-1374 [~]$ kubectl get pods -n openebs -l app=cstor-pool\n NAME READY STATUS RESTARTS AGE\n cstor-disk-pool-rgy3-57f965b48c-srz2x 3/3 Running 0 8m33s\n cstor-disk-pool-gij4-77bb4b8f44-s6k89 3/3 Running 0 8m33s\n cstor-disk-pool-09l1-56d444996b-m698h 3/3 Running 0 8m33s\n\nAfter creating cStor pool, we can proceed to create volume. For illustration, let us deploy a busybox with cstor volume as its persistent storage. Before creating a pool, we need to create a storage class specifying the storagePoolClaim that was created in the above step as follows:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-cstor\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk-pool\"\n - name: ReplicaCount\n value: \"3\"\n provisioner: openebs.io/provisioner-iscsi\n\nPopulate storagePoolClaim and ReplicaCount as per your requirement in the above definition and create storage class. You have to use this storage class while creating PVC.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: openebs-pvc\n spec:\n storageClassName: openebs-cstor\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10Gi\n ---\n apiVersion: v1\n kind: Service\n metadata:\n labels:\n name: busybox\n name: busybox\n spec:\n clusterIP: None\n selector:\n app: busybox\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: busybox\n labels:\n app: busybox\n spec:\n selector:\n matchLabels:\n app: busybox\n template:\n metadata:\n labels:\n app: busybox\n spec:\n containers:\n - name: app-busybox\n imagePullPolicy: IfNotPresent\n image: busybox\n command: [\"/bin/sh\"]\n args: [\"-c\", \"while true; do sleep 10;done\"]\n env:\n volumeMounts:\n - name: data-vol\n mountPath: /busybox\n volumes:\n - name: data-vol\n persistentVolumeClaim:\n claimName: openebs-pvc\n\nAfter updating the storage class in the above manifest, let us deploy the busybox application by applying the above definition.\n\nCheck if the PVC is created and mounted successfully on the application pod by checking their status.\n\n k3os-1374 [~]$ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n openebs-pvc Bound pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d 10Gi RWO openebs-cstor 4m35s\n \n k3os-1374 [~]$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n busybox-748fb77c75-9lwzz 1/1 Running 0 4m42s\n\nAs we specified ReplicaCount as 3 in the storage class, 3 volume replicas will be created which can be found by executing the following command.\n\n k3os-1374 [~]$ kubectl get cvr -n openebs\n NAME USED ALLOCATED STATUS AGE\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-09l1 7.95M 116K Healthy 6m37s\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-rgy3 7.95M 116K Healthy 6m37s\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-gij4 7.95M 116K Healthy 6m37s\n\nThus, the cStor engine can be used to provision persistent volume for the workloads in K3s.\n\nPlease leave your valuable comments or feedback in the comment section below if you find this tutorial helpful.\n","notHasFeatureImage":false,"slug":"getting-started-with-k3s-in-vsphere-and-openebs-cstor"},{"id":32,"title":"Resizing the ZFS-LocalPV Volumes","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"26-03-2020","tags":["LocalPV","OpenEBS"],"excerpt":"In this post, we will focus on how we can resize the volumes provisioned by ZFS-LocalPV without restarting the application.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can resize the volumes provisioned by ZFS-LocalPV without restarting the application.\n\n### **Prerequisite**\n\nPlease make sure you have installed the ZFS-LocalPV Driver version v0.5 or later:\n\n $ kubectl apply -f\n https://raw.githubusercontent.com/openebs/zfs-localpv/v0.5.x/deploy/zfs-operator.yaml\n\nMake sure you are using k8s version 1.16+ as this feature is in beta. In Kubernetes 1.14 and 1.15, this feature was in alpha status and required enabling the following feature gate:\n\n --feature-gates=ExpandCSIVolumes=true\n\nAlso for Kubernetes 1.14 and 1.15, online expansion feature gate has to be enabled explicitly:\n\n --feature-gates=ExpandInUsePersistentVolumes=true\n\n### **Introduction**\n\nThe ZFS-LocalPV CSI driver supports ONLINE volume expansion, which means, if the application is using the volume, you can perform the volume expansion. This also means that the volume expansion will be performed when an application is using that volume. So, if an application is not running and we have performed the resize operation, the Driver will wait for the application to start for the resize operation to complete.\n\n### **Setup**\n\nCreate the StorageClass with allowVolumeExpansion as true. We can resize only those volumes which are using the StorageClass with this flag as true.\n\n $ cat sc.yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n $ kubectl apply -f sc.yaml\n storageclass.storage.k8s.io/openebs-zfspv created\n\nCreate the PVC using the above StorageClass:\n\n $ cat pvc.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: csi-zfspv\n spec:\n storageClassName: openebs-zfspv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n $ kubectl apply -f pvc.yaml\n persistentvolumeclaim/csi-zfspv created\n\nNow deploy the application using the above PVC. Here, we will be using below Percona application:\n\n $ cat percona.yaml\n apiVersion: v1\n kind: ConfigMap\n metadata:\n annotations:\n name: sqltest\n namespace: default\n data:\n sql-test.sh: |\n #!/bin/bash\n DB_PREFIX=\"Inventory\"\n DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`\n DB_NAME=\"${DB_PREFIX}_${DB_SUFFIX}\"\n echo -e \"\\nWaiting for mysql server to start accepting connections..\"\n retries=10;wait_retry=30\n for i in `seq 1 $retries`; do\n mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1\n rc=$?\n [ $rc -eq 0 ] && break\n sleep $wait_retry\n done\n if [ $rc -ne 0 ];\n then\n echo -e \"\\nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exiting\\n\"\n exit 1\n fi\n mysql -uroot -pk8sDem0 -e \"CREATE DATABASE $DB_NAME;\"\n mysql -uroot -pk8sDem0 -e \"CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"INSERT INTO Hardware (id, name, owner, description) values (1, \"dellserver\", \"basavaraj\", \"controller\");\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"DROP DATABASE $DB_NAME;\"\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: percona\n labels:\n name: percona\n spec:\n replicas: 1\n selector:\n matchLabels:\n name: percona\n template:\n metadata:\n labels:\n name: percona\n spec:\n containers:\n - resources:\n name: percona\n image: openebs/tests-custom-percona:latest\n imagePullPolicy: IfNotPresent\n args:\n - \"--ignore-db-dir\"\n - \"lost+found\"\n env:\n - name: MYSQL_ROOT_PASSWORD\n value: k8sDem0\n ports:\n - containerPort: 3306\n name: percona\n volumeMounts:\n - mountPath: /var/lib/mysql\n name: demo-vol1\n - mountPath: /sql-test.sh\n subPath: sql-test.sh\n name: sqltest-configmap\n livenessProbe:\n exec:\n command: [\"bash\", \"sql-test.sh\"]\n initialDelaySeconds: 30\n periodSeconds: 1\n timeoutSeconds: 10\n volumes:\n - name: demo-vol1\n persistentVolumeClaim:\n claimName: csi-zfspv\n - name: sqltest-configmap\n configMap:\n name: sqltest\n ---\n apiVersion: v1\n kind: Service\n metadata:\n name: percona-mysql\n labels:\n name: percona-mysql\n spec:\n ports:\n - port: 3306\n targetPort: 3306\n selector:\n name: percona\n\nApply the above YAML to deploy the Percona application:\n\n $ kubectl apply -f percona.yaml\n configmap/sqltest created\n deployment.apps/percona created\n service/percona-mysql created\n\nNow the setup is ready and the application is running:\n\n $ kubectl get po\n NAME READY STATUS RESTARTS AGE\n percona-9449b4b9c-48qpw 1/1 Running 0 38s\n\nCheck the volume size at the application size:\n\n $ kubectl exec -it percona-9449b4b9c-48qpw bash\n root@percona-9449b4b9c-48qpw:/# df -h\n Filesystem Size Used Avail Use% Mounted on\n none 91G 18G 69G 21% /\n tmpfs 3.9G 0 3.9G 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sda1 91G 18G 69G 21% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n /dev/zd0 4.9G 234M 4.7G 5% /var/lib/mysql\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nFrom above o/p we can see that the volume has been created of size 5Gi and it is attached to the application at the given mount point (/var/lib/mysql).\n\n### **Volume Resize**\n\nHere, we just have to update the PVC with the new size and apply it. Please note that volume shrinking is not supported, so you have to change the size to a higher value. Here, in our case, we will update the size to 8Gi\n\n $ cat pvc.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: csi-zfspv\n spec:\n storageClassName: openebs-zfspv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 8Gi\n\nApply the above YAML to perform the resize:\n\n $ kubectl apply -f pvc.yaml\n persistentvolumeclaim/csi-zfspv configured\n\nNow, we can keep checking the PVC for the new size to be updated, it may take a while. Once resize operation is done we can see the PVC output with the updated size:\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-9b5c22a5-29be-428e-aa96-5e183c1c4c62 8Gi RWO openebs-zfspv 33m\n\nWe can also exec into the application pod and verify that the new size is visible to the application:\n\n $ kubectl exec -it percona-9449b4b9c-48qpw bash\n root@percona-9449b4b9c-48qpw:/# df -h\n Filesystem Size Used Avail Use% Mounted on\n none 91G 18G 69G 21% /\n tmpfs 3.9G 0 3.9G 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sda1 91G 18G 69G 21% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n /dev/zd0 7.9G 237M 7.6G 3% /var/lib/mysql\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nI hope you find this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"resizing-the-zfs-localpv-volumes"},{"id":33,"title":"Snapshot and Clone for ZFS LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"03-03-2020","tags":["OpenEBS","ZFS","Open Source"],"excerpt":"In this post, we will focus on how we can create a snapshot and clone for volumes provisioned by ZFS-LocalPV.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.2b738ae93497639f7465a332e1aef247.1584602510099.1584602510099.1584602510099.1&__hssc=216392137.1.1584602510100&__hsfp=2870217423) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can create a snapshot and clone for volumes provisioned by ZFS-LocalPV.\n\n#### **Prerequisite**\n\nFor clone, we need to have VolumeSnapshotDataSource support, which is in beta in Kubernetes 1.17. If you are using the Kubernetes version less than 1.17, you have to enable the VolumeSnapshotDataSource feature gate at kubelet and kube-apiserver.\n\n#### **Snapshot**\n\nWe can create a snapshot of a volume that can be used further for creating a clone and for taking a backup. To create a snapshot, we have to first create a SnapshotClass just like a storage class where you can provide deletionPolicy as Retain or Delete.\n\n $ cat snapshotclass.yaml\n kind: VolumeSnapshotClass\n apiVersion: snapshot.storage.k8s.io/v1beta1\n metadata:\n name: zfspv-snapclass\n annotations:\n snapshot.storage.kubernetes.io/is-default-class: \"true\"\n driver: zfs.csi.openebs.io\n deletionPolicy: Delete\n\nApply the snapshotclass YAML:\n\n $ kubectl apply -f snapshotclass.yaml\n volumesnapshotclass.snapshot.storage.k8s.io/zfspv-snapclass created\n\nFind a PVC for which snapshot has to be created\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 4Gi RWO openebs-zfspv 2m35s\n\nCreate the snapshot using the created SnapshotClass for the selected PVC\n\n $ cat snapshot.yaml\n apiVersion: snapshot.storage.k8s.io/v1beta1\n kind: VolumeSnapshot\n metadata:\n name: zfspv-snap\n spec:\n volumeSnapshotClassName: zfspv-snapclass\n source:\n persistentVolumeClaimName: csi-zfspv\n\nApply the snapshot.yaml\n\n $ kubectl apply -f snapshot.yaml\n volumesnapshot.snapshot.storage.k8s.io/zfspv-snap created\n\nPlease note that you have to create the snapshot in the same namespace where the PVC is created. Check the created snapshot resource, make sure readyToUsefield is true, before using this snapshot for any purpose.\n\n $ kubectl get volumesnapshot.snapshot\n NAME AGE\n zfspv-snap 2m8s\n \n $ kubectl get volumesnapshot.snapshot zfspv-snap -o yaml\n apiVersion: snapshot.storage.k8s.io/v1beta1\n kind: VolumeSnapshot\n metadata:\n annotations:\n kubectl.kubernetes.io/last-applied-configuration: |\n {\"apiVersion\":\"snapshot.storage.k8s.io/v1beta1\",\"kind\":\"VolumeSnapshot\",\"metadata\":{\"annotations\":{},\"name\":\"zfspv-snap\",\"namespace\":\"default\"},\"spec\":{\"source\":{\"persistentVolumeClaimName\":\"csi-zfspv\"},\"volumeSnapshotClassName\":\"zfspv-snapclass\"}}\n creationTimestamp: \"2020-02-25T08:25:51Z\"\n finalizers:\n - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection\n - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\n generation: 1\n name: zfspv-snap\n namespace: default\n resourceVersion: \"447494\"\n selfLink: /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/zfspv-snap\n uid: 3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n spec:\n source:\n persistentVolumeClaimName: csi-zfspv\n volumeSnapshotClassName: zfspv-snapclass\n status:\n boundVolumeSnapshotContentName: snapcontent-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n creationTime: \"2020-02-25T08:25:51Z\"\n readyToUse: true\n restoreSize: \"0\"\n\nCheck the OpenEBS resource for the created snapshot. Check, status should be Ready.\n\n $ kubectl get zfssnap -n openebs\n NAME AGE\n snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 3m32s\n \n $ kubectl get zfssnap snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd -n openebs -oyaml\n apiVersion: openebs.io/v1alpha1\n kind: ZFSSnapshot\n metadata:\n creationTimestamp: \"2020-02-25T08:25:51Z\"\n finalizers:\n - zfs.openebs.io/finalizer\n generation: 2\n labels:\n kubernetes.io/nodename: e2e1-node2\n openebs.io/persistent-volume: pvc-73402f6e-d054-4ec2-95a4-eb8452724afb\n name: snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n namespace: openebs\n resourceVersion: \"447328\"\n selfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/zfssnapshots/snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n uid: 6142492c-3785-498f-aa4a-569ec6c0e2b8\n spec:\n capacity: \"4294967296\"\n fsType: zfs\n ownerNodeID: e2e1-node2\n poolName: test-pool\n volumeType: DATASET\n status:\n state: Ready\n\nWe can go to the node and confirm that snapshot has been created:\n\n # zfs list -t all\n NAME USED AVAIL REFER MOUNTPOINT\n test-pool 818K 9.63G 24K /test-pool\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 24K 4.00G 24K /var/lib/kubelet/pods/3862895a-8a67-446e-80f7-f3c18881e391/volumes/kubernetes.io~csi/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb/mount\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 0B - 24K -\n\n#### **Clone**\n\nWe can create a clone volume from a snapshot and use that volume for some application. We can create a PVC YAML and mention the snapshot name in the datasource.\n\n $ cat clone.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: zfspv-clone\n spec:\n storageClassName: openebs-zfspv\n dataSource:\n name: zfspv-snap\n kind: VolumeSnapshot\n apiGroup: snapshot.storage.k8s.io\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nThe above YAML says that create a volume from the snapshot zfspv-snap. Applying the above YAML will create a clone volume on the same node where the original volume is present. The newly created clone PV will also be there on the same node where the original PV is there. Apply the clone YAML\n\n $ kubectl apply -f clone.yaml \n persistentvolumeclaim/zfspv-clone created\n\nNote that the clone PVC should also be of the same size as that of the original volume. Currently resize is not supported. Also, note that the poolname should also be same, as across the ZPOOL clone is not supported. So, if you are using a separate storageclass for the clone PVC, please make sure it refers to the same ZPOOL.\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 4Gi RWO openebs-zfspv 13m\n zfspv-clone Bound pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 4Gi RWO openebs-zfspv 34s\n\nWe can see in the above output that zfspv-clone claim has been created and it is bound. Also, we can check the zfs list on node and verify that clone volume is created.\n\n $ zfs list -t all\n NAME USED AVAIL REFER MOUNTPOINT\n test-pool 834K 9.63G 24K /test-pool\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 24K 4.00G 24K /var/lib/kubelet/pods/3862895a-8a67-446e-80f7-f3c18881e391/volumes/kubernetes.io~csi/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb/mount\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 0B - 24K -\n test-pool/pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 0B 9.63G 24K none\n\nThe clone volume will have properties same as snapshot properties which are the properties when that snapshot has been created. The ZFSVolume object for the clone volume will be something like below:\n\n $ kubectl describe zv pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 -n openebs\n Name: pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19\n Namespace: openebs\n Labels: kubernetes.io/nodename=e2e1-node2\n Annotations: none\n API Version: openebs.io/v1alpha1\n Kind: ZFSVolume\n Metadata:\n Creation Timestamp: 2020-02-25T08:34:25Z\n Finalizers:\n zfs.openebs.io/finalizer\n Generation: 1\n Resource Version: 448930\n Self Link: /apis/openebs.io/v1alpha1/namespaces/openebs/zfsvolumes/pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19\n UID: e38a9f9a-fb76-466b-a6f9-8d070e0bec6f\n Spec:\n Capacity: 4294967296\n Fs Type: zfs\n Owner Node ID: e2e1-node2\n Pool Name: test-pool\n Snapname: pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n Volume Type: DATASET\n Events: none\n\nHere you can note that this resource has Snapname field which tells that this volume is created from that snapshot.\n\nI hope you found this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"snapshot-and-clone-for-zfs-localpv"},{"id":34,"title":"Setup Continuous Integration for Helm chart","author":"Intakhab Ali","author_info":"Software Engineer at MayaData","date":"05-02-2020","tags":["Helm","OpenEBS","Tutorials","Kubernetes"],"excerpt":"In this blog, we'll set up a continuous integration of the Helm chart. We'll package the Helm chart with the help of CI tools & push them to chart registry.","content":"\n[Helm](https://www.helm.sh/) is a package manager for Kubernetes that allows developers and operators to easily package, configure, and deploy applications and services onto Kubernetes clusters.\n\nHelm is now an official Kubernetes project and is part of the[ Cloud Native Computing Foundation](https://www.cncf.io/), a non-profit Linux Foundation that supports Open Source projects in and around the Kubernetes ecosystem.\n\nIn this tutorial, we will set up a continuous integration of the Helm chart. We will package the Helm chart with the help of CI tools like (Travis, Jenkins), and push it to chart registries like (Harbor, Chartmuseum).\n\n## Prerequisites:\n\n- Registry to store Helm like Harbor or Chartmuseum\n- Understanding of Helm and any of the CI platforms (Travis, Jenkins, circle, CI)\n- A Git repository to maintain version control of helm chart\n\n**I am going to use Travis as a CI platform and Harbor as a Helm registry to host the helm.**\n\n**As I choose Travis here, .travis.yml consists of the job lifecycle. Let’s write job cycle for the helm chart.**\n\n### Lifecycle 1:\n\nChoose the base language as Python\n\n ---\n language: python\n\nWe need to have some environment variables so that we can update whenever there’s a new version of the Helm release or change of the registry URL.\n\nHere is the list of a variable that we’ll need:\n\n***HELM_URL=[https://storage.googleapis.com/kubernetes-helm](https://storage.googleapis.com/kubernetes-helm) (this is the URL where we can download the helm package)***\n\n***HELM_TGZ=helm-v2.4.2-linux-amd64.tar.gz (this is the Helm tar filename)***\n\n***REPO_DIR=/home/travis/build/inyee786/test-helm (this is the path where Travis keep Git folder)***\n\n***YAMLLINT_VERSION=1.8.1 (this is yamllint version which is used to check lint of file)***\n\n***HARBOR_CHART_URL=https://harbor-test.mayadata.io/chartrepo (change this according to your chart registry url{harbor or Chartmuseum}***\n\n***HARBOR_PROJECT_NAME=maya (this is the Harbor project name, where we will store the chart)***\n\n***CHART_FOLDER=charts (this is the folder name, where we can keep the Helm charts)***\n\nIt looks like this\n\n env:\n global:\n - HELM_URL=https://storage.googleapis.com/kubernetes-helm\n - HELM_TGZ=helm-v2.4.2-linux-amd64.tar.gz\n - REPO_DIR=/home/travis/build/inyee786/test-helm\n - YAMLLINT_VERSION=1.8.1\n - HARBOR_CHART_URL=https://harbor-\n test.mayadata.io/chartrepo\n - HARBOR_PROJECT_NAME=maya\n - CHART_FOLDER=charts\n\nWe need some private variables, where we can store the credentials and push it to the Helm registry (Harbor has an excellent feature where we can have bot user, and you can use the bot credential). All we have to feed is-\n\n HARBOR_USERNAME:\n HARBOR_PASSWORD:\n\nInside Travis, go to (****settings > Environment Variables****) to set the private env\n\n![Environment variables](/images/blog/environment-variables.png)\n\n### Lifecycle 2 :\n\nInstall the Prerequisites to Set up a CI environment to build and check the YAML lint.\n\nDownload helm and ****untar**** the chart after downloading\n\n- wget ${HELM_URL}/${HELM_TGZ}\n- tar xzfv ${HELM_TGZ}\n- PATH=`pwd`/linux-amd64/:$PATH\n\nInitialize the helm client and update the helm repo\n\n- Helm init — client-only\n- Helm repo update\n\nInstall helm plugin to push chart on the registry\n\n- Helm plugin install[ https://github.com/chartmuseum/helm-push](https://github.com/chartmuseum/helm-push) — version v0.7.1\n\nInstall yamllint python package to check the lint\n\n- sudo pip install yamllint==”${YAMLLINT_VERSION}”\n\nIt looks like the below config\n\n install:\n # Installing Helm\n - wget ${HELM_URL}/${HELM_TGZ}\n - tar xzfv ${HELM_TGZ}\n - PATH=`pwd`/linux-amd64/:$PATH\n - helm init --client-only\n # helm plugin to push helm chart\n - helm plugin install https://github.com/chartmuseum/helm-\n push --version v0.7.1\n # Installing pip deps\n - sudo pip install yamllint==\"${YAMLLINT_VERSION}\"\n - helm repo update\n\n### Lifecycle 3 :\n\nBefore going further to build a chart, we need to run some script to check the lint in the chart and Travis file. It is a good practice to check the lint\n\nCheck the Helm lint of all Helm chart\n\n- For dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\nhelm lint ${REPO_DIR}/${CHART_FOLDER}/$dir\nif [ $? != 0 ]; then\ntravis_terminate 1\nfi\n\nTo check the YAML lint for travis.yml, chart.yaml and value.yaml, we use the yamllint python package. We need the rule to check the lint.\n\n- yamllint -c .yamllint.yml -s .travis.yml .yamllint.yml\n- yamllint -c .yamllint.yml -s $(find . -type f -name “Chart.yaml”)\n- yamllint -c .yamllint.yml -s $(find . -type f -name “values.yaml”)\n\nThe script section should look like the below config\n\n script:\n # Check charts format\n - >\n for dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\n helm lint ${REPO_DIR}/${CHART_FOLDER}/$dir\n if [ $? != 0 ]; then\n travis_terminate 1\n fi\n done\n # Check YAML styling\n - yamllint -c .yamllint.yml -s .travis.yml .yamllint.yml\n - yamllint -c .yamllint.yml -s $(find . -type f -name \"Chart.yaml\")\n - yamllint -c .yamllint.yml -s $(find . -type f -name \"values.yaml\")\n\nHere comes the interesting part where we are going to build and package the chart.\n\n****Lifecycle 4:****\n\nIt’s better to build and push when we merge the chart in the **master** branch. So we run the below command when we merge the chart in the **master** branch\n\nWe need a temporary directory where we will build and package the chart\n\n- BUILD_DIR=$(mktemp -d)\n\nRun a loop to all the charts to build, package, and push it to the registry. The below commands will run on each chart\n\n- helm dep update ${REPO_DIR}/${CHART_FOLDER}/$dir\n\nPackage the chart with the below command\n\n- helm package ${REPO_DIR}/${CHART_FOLDER}/$dir\n\nThen push the chart to registry\n\n- helm push — username ${HARBOR_USERNAME} — password ${HARBOR_PASSWORD} ${REPO_DIR}/${CHART_FOLDER}/$dir ${HARBOR_CHART_URL}/maya\n\nBelow is the what the config will look like\n\n # Temporary dir for storing new packaged charts and index files\n BUILD_DIR=$(mktemp -d) # Push temporary directory to the stack\n pushd $BUILD_DIR # Iterate over all charts are package them push it to Harbor\n for dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\n helm dep update ${REPO_DIR}/${CHART_FOLDER}/$dir\n helm package ${REPO_DIR}/${CHART_FOLDER}/$dir\n helm push --username ${HARBOR_USERNAME} --password ${HARBOR_PASSWORD} ${REPO_DIR}/${CHART_FOLDER}/$dir ${HARBOR_CHART_URL}/maya\n if [ $? != 0 ]; then\n travis_terminate 1\n fi\n done# Pop temporary directory from the stack\n popd\n\nWow! We have successfully completed all the steps. Now, our setup is ready to build and push the helm chart to the registry.\n\n![Project Dashboard](/images/blog/project-dashboard.png)\n\nHere is the full Travis file\n[https://gist.github.com/inyee786/d779f347d7fa272aed4ee8457182af35.js](https://gist.github.com/inyee786/d779f347d7fa272aed4ee8457182af35.js)\n\nHere is .yamllint.yml file which contains lint rule for charts.yaml values.yaml and .travis.yaml\n[https://gist.github.com/inyee786/ef15b05c98bb4761b41af5f4fe268239.js](https://gist.github.com/inyee786/ef15b05c98bb4761b41af5f4fe268239.js)\n\n## Conclusion:\n\nHere we packaged the helm chart and pushed it to the helm registry.\n\n## About me\n\nYou can follow me at the below profiles and can ask any questions related to Angular, JavaScript, Travis, Kubernetes, etc.\n\n- [GitHub](https://github.com/inyee786/)\n- [Linkedin](https://www.linkedin.com/in/intakhab-ali/)\n- [Medium](https://medium.com/@intakhab.cusat)\n\nThis blog was originally published on 28th Jan 2020 on [MayaData’s blog](https://blog.mayadata.io/openebs/setup-continuous-integration-for-helm-chart).\n","notHasFeatureImage":false,"slug":"setup-continuous-integration-for-helm-chart"},{"id":35,"title":"Monitoring ZFS-LocalPV Volumes","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"28-01-2020","tags":["OpenEBS","LocalPV","ZFS"],"excerpt":"In this post, we will focus on how we can set up the Prometheus alert for Provisioned volumes when space utilization has reached a critical point.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can set up the Prometheus alert for Provisioned volumes when space utilization has reached a critical point.\n\n### Prerequisite\n\nMake sure you are using k8s version 1.15+ to access the CSI volume metrics.\n\n### Setup helm\n\nThis step uses helm as the Kubernetes package manager. If you have not setup the helm, execute the below configuration. Otherwise, you can move on to the next step.\n\n $ helm version\n Client: &version.Version{SemVer:\"v2.16.1\", GitCommit:\"bbdfe5e7803a12bbdf97e94cd847859890cf4050\", GitTreeState:\"clean\"}\n Server: &version.Version{SemVer:\"v2.16.1\", GitCommit:\"bbdfe5e7803a12bbdf97e94cd847859890cf4050\", GitTreeState:\"clean\"}\n \n $ helm init\n Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n \n Please note: By default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n To prevent this, run `helm init` with the --tiller-tls-verify flag.\n For more information on securing your installation see: (https://docs.helm.sh/using_helm/#securing-your-helm-installation)[https://docs.helm.sh/using_helm/#securing-your-helm-installation]\n \n $ kubectl create serviceaccount --namespace kube-system tiller\n serviceaccount/tiller created\n \n $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller\n clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created\n \n $ kubectl patch deploy --namespace kube-system tiller-deploy -p '{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}'\n deployment.extensions/tiller-deploy patched\n\n### Install Prometheus Operator\n\nOnce the helm is ready and the related tiller pods are up and running, use the Prometheus chart from the helm repository.\n\n $ helm install stable/prometheus-operator --name prometheus-operator\n\nCheck all the required pods are up and running\n\n $ kubectl get pods -l \"release=prometheus-operator\"\n NAME READY STATUS RESTARTS AGE\n prometheus-operator-grafana-85bb5d49d-bffdg 2/2 Running 0 2m21s\n prometheus-operator-operator-64844759f7-rpwws 2/2 Running 0 2m21s\n prometheus-operator-prometheus-node-exporter-p9rl8 1/1 Running 0 2m21s\n\n### Set up the alert rule\n\nCheck all the rules available in the system:\n\n $ kubectl get PrometheusRule\n NAME AGE\n prometheus-operator-alertmanager.rules 4m21s\n prometheus-operator-etcd 4m21s\n prometheus-operator-general.rules 4m21s\n prometheus-operator-k8s.rules 4m21s\n prometheus-operator-kube-apiserver-error 4m21s\n prometheus-operator-kube-apiserver.rules 4m21s\n prometheus-operator-kube-prometheus-node-recording.rules 4m21s\n prometheus-operator-kube-scheduler.rules 4m21s\n prometheus-operator-kubernetes-absent 4m21s\n prometheus-operator-kubernetes-apps 4m21s\n prometheus-operator-kubernetes-resources 4m21s\n prometheus-operator-kubernetes-storage 4m21s\n prometheus-operator-kubernetes-system 4m21s\n prometheus-operator-kubernetes-system-apiserver 4m21s\n prometheus-operator-kubernetes-system-controller-manager 4m21s\n prometheus-operator-kubernetes-system-kubelet 4m21s\n prometheus-operator-kubernetes-system-scheduler 4m21s\n prometheus-operator-node-exporter 4m21s\n prometheus-operator-node-exporter.rules 4m21s\n prometheus-operator-node-network 4m21s\n prometheus-operator-node-time 4m21s\n prometheus-operator-node.rules 4m21s\n prometheus-operator-prometheus 4m21s\n prometheus-operator-prometheus-operator 4m21s\n\nYou can edit any of the default rules or create a new rule to get the alerts. Below is a sample rule to start generating alerts when available storage space is less than 10%.\n\n apiVersion: monitoring.coreos.com/v1\n kind: PrometheusRule\n metadata:\n labels:\n app: prometheus-operator\n chart: prometheus-operator-8.5.4\n heritage: Tiller\n release: prometheus-operator\n name: prometheus-operator-zfs-alertmanager.rules\n namespace: default\n spec:\n groups:\n - name: zfsalertmanager.rules\n rules:\n - alert: ZFSVolumeUsageCritical\n annotations:\n message: The PersistentVolume claimed by {{ $labels.persistentvolumeclaim\n }} in Namespace {{ $labels.namespace }} is only {{ printf \"%0.2f\" $value\n }}% free.\n expr: |\n 100 * kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n /\n kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n < 10\n for: 1m\n labels:\n severity: critical\n\nNow apply the above YAML so that Prometheus can fire the alerts when available space is less than 10%.\n\n### Check the Prometheus alert\n\nTo view the Prometheus web UI, you must expose it through a Service. A simple way to accomplish this is to use a Service of type NodePort.\n\n $ cat prometheus-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n name: prometheus-service\n spec:\n type: NodePort\n ports:\n - name: web\n nodePort: 30090\n port: 9090\n protocol: TCP\n targetPort: web\n selector:\n prometheus: prometheus-operator-prometheus\n\nApply the above YAML\n\n $ kubectl apply -f prometheus-service.yaml\n service/prometheus-service created\n\nNow you can access the alert manager UI via “node’s-external-ip:30090”.\n\n $ kubectl get nodes -owide\n NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\n gke-zfspv-pawan-default-pool-3e407350-xvzp Ready 103m v1.15.4-gke.22 10.168.0.45 34.94.3.140 Ubuntu 18.04.3 LTS 5.0.0-1022-gke docker://19.3.2\n\nHere, we can access the alert manager via URL: [http://34.94.3.140:30090/](http://34.94.3.140:30090/)\n\n### Check the Alert Manager\n\nTo view the Alert Manager web UI, expose it through a Service of type NodePort.\n\n $ cat alertmanager-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n name: alertmanager-service\n spec:\n type: NodePort\n ports:\n - name: web\n nodePort: 30093\n port: 9093\n protocol: TCP\n targetPort: web\n selector:\n alertmanager: prometheus-operator-alertmanager\n\nApply the above YAML\n\n $ kubectl apply -f alertmanager-service.yaml\n service/alertmanager-service created\n\nNow you can access the alert manager UI via “node’s-external-ip:30093”.\n\n $ kubectl get nodes -owide\n NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\n gke-zfspv-pawan-default-pool-3e407350-xvzp Ready 103m v1.15.4-gke.22 10.168.0.45 34.94.3.140 Ubuntu 18.04.3 LTS 5.0.0-1022-gke docker://19.3.2\n\nAgain, we can access the alert manager via URL: [http://34.94.3.140:30093/.](http://34.94.3.140:30093/)\n\nI hope you found this post to be useful. Feel free to contact me with any feedback or questions by using the comment section below.\n\nThis blog was originally published on [Jan 22, 2020, on the MayaData blog](https://blog.mayadata.io/openebs/monitoring-zfs-localpv-volumes).\n","notHasFeatureImage":false,"slug":"monitoring-zfs-localpv-volumes"},{"id":36,"title":"Uniquely identifying disks in OpenEBS on VMWare platform","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"22-01-2020","tags":["OpenEBS","Vmware","Vcenter","Virtual Disk","Kubernetes"],"excerpt":"A little bit of background. I work at a company called MayaData who develops a very cool Open Source software called OpenEBS (CNCF Sandbox project) that simplifies the deployment of stateful applications on Kubernetes.","content":"\nA little bit of background. I work at a company called [MayaData](https://mayadata.io/) who develops a very cool Open Source software called OpenEBS (CNCF Sandbox project) that simplifies the deployment of stateful applications on Kubernetes. You should check it out at [www.openebs.io](http://www.openebs.io/?__hstc=216392137.84d52389458ef57b0491fddb252202d6.1570688281471.1578466343199.1578469779597.19&__hssc=216392137.2.1578469779597&__hsfp=2854279793).\n\nKubernetes can be installed on any type of machine; be it a Virtual Machine, bare metal, or cloud machine. Kubernetes abstracts away most of the significant bits of a system, except storage. When it comes to storage, the main reason an abstraction will not work is that there is no uniqueness among the storage devices themselves. Every vendor and every virtualization platform implements it differently.\n\nWe hit this issue of unique virtual disks while deploying OpenEBS on Kubernetes backed by VMware VMS or other virtualization platforms because OpenEBS NDM is not able to uniquely identify the block devices themselves.\n\nChanging the absolute configuration on the Virtual Machine can help you get around this issue.\n\nHere are the steps to enable unique disk IDs in VMware via vSphere client:\n\n1. Right-click the virtual machine for which you want to enable the disk UUID attribute, and select Power > Power Off.\n2. The virtual machine powers off.\n3. Right-click the virtual machine, and click Edit Settings.\n4. Click the Options tab, and select the General entry in the settings column.\n5. Click Configuration Parameters. The Configuration Parameters window appears.\n6. Click Add Row.\n7. In the Name column, enter disk.Enable UUID\n8. In the Value column, enter TRUE.\n9. Click OK and click Save.\n10. Power on the virtual machine.\n\nThis will assign WWN to each disk in the Virtual Machine\n\nThat is it for today’s tutorial. If you have any questions, feedback, or any topic that you feel I should cover next, feel free to comment on our blog or reach out to us on our [Slack](https://slack.openebs.io./) channel.\n\nThis blog was originally published on [Oct 01, 2019, on the MayaData blog](https://blog.mayadata.io/openebs/uniquely-identifying-disks-in-openebs-on-vmware-platform).\n","notHasFeatureImage":false,"slug":"uniquely-identifying-disks-in-openebs-on-vmware-platform"},{"id":37,"title":"Creating manual BlockDevice","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"16-01-2020","tags":["Block Devices","Docker","Uncategorized","Disk","OpenEBS","Kubernetes"],"excerpt":"BlockDevices are the consumable units of storage in the OpenEBS storage solution. Users can create BlockDevices manually to support custom partitions/lvms, etc., which are not detected by NDM. To create a manual BlockDevice, follow the steps below","content":"\nBlockDevices are the consumable units of storage in the OpenEBS storage solution. Currently, NDM supports the discovery and management of only a complete disk.\n\nHowever, users can create BlockDevices manually to support custom partitions/lvms, etc., which are not detected by NDM. To create a manual BlockDevice, follow the steps below:\n\n1. Download the sample block device custom resource YAML file. \n ```\n wget\n https://raw.githubusercontent.com/openebs/node-disk-manager/master/deploy/crds/openebs_v1alpha1_blockdevice_cr.yaml\n ```\n\n2.  Edit the file and fill in the details of the blockdevice. Fields marked with optional are not mandatory and can be removed. All other fields are required and information provided will be used while claiming.\n ```\n apiVersion: openebs.io/v1alpha1\n kind: BlockDevice\n metadata:\n name: example-blockdevice\n labels:\n kubernetes.io/hostname: # like gke-openebs-user-default-pool-044afcb8-bmc0\n ndm.io/managed: \"false\" # for manual disk creation put false\n ndm.io/blockdevice-type: blockdevice\n status:\n claimState: Unclaimed\n state: Active\n spec:\n capacity:\n storage: #like 53687091200\n details:\n firmwareRevision: #optional\n model: # like PersistentDisk, optional\n serial: # like google-disk-2, optional\n compliance: #like \"SPC-4\", optional\n vendor: #like Google, optional\n devlinks:\n - kind: by-id\n links:\n - # like /dev/disk/by-id/scsi-0Google_PersistentDisk_disk-2\n - # like /dev/disk/by-id/google-disk-2\n - kind: by-path\n links:\n - # like /dev/disk/by-path/virtio-pci-0000:00:03.0-scsi-0:0:2:0\n nodeAttributes:\n nodeName: # output of `kubectl get nodes` can be used\n path: # like /dev/sdb\n ```\n3.  Apply the YAML file.\n ```\n kubectl apply -f openebs_v1alpha1_blockdevice_cr.yaml\n ```\n\nThe BlockDevice CR will be created and is then used by NDM Operator for claiming, but it won’t be managed by NDM Daemon for any changes that happen on the device. However, all the Claim/Unclaim operations and cleanup operations will be performed on this BlockDevice.\n\nPlease provide your valuable feedback & comments below and let me know what I can cover in my next blog.\n\nThis blog was originally published on Oct 22nd, 2019, on the [MayaData blog](https://blog.mayadata.io/openebs/creating-manual-blockdevice).\n","notHasFeatureImage":false,"slug":"creating-manual-blockdevice"},{"id":38,"title":"OpenEBS Node Device Management (NDM) — Troubleshooting tips","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"08-01-2020","tags":["Docker","OpenEBS","Uncategorized","Troubleshooting","Tutorials"],"excerpt":"OpenEBS Node Device Management (aka NDM) helps in discovering the block devices attached to Kubernetes nodes. In this blog, I will walk through some of the scenarios I have seen working with users on the OpenEBS Slack Channel.","content":"\nOpenEBS Node Device Management (aka NDM) helps in discovering the block devices attached to Kubernetes nodes. For many clusters, the default configuration of NDM suffices, however there are some cases where further customizations are required.\n\n> In this blog, I will walk through some of the scenarios I have seen working with Users on the [OpenEBS Slack Channel](http://slack.openebs.io/)\n\n---\n\n### NDM Quick Overview\n\nFor setting up NDM in secure mode, please see my previous [blog](https://blog.mayadata.io/openebs/configuring-openebs-to-run-with-security-enhanced-linux), and you can learn how NDM works [here](/docs/user-guides/ndm). Here is a quick snapshot of the key components of NDM.\n\n- NDM components are installed in the OpenEBS Namespace. Ensure that NDM pods part of the NDM DaemonSet are running on all the storage nodes. NDM Operator helps with allocating Block Devices to Block Device Claims and should be running.\n- NDM DaemonSet pod discovers all the block devices attached to the node and creates BlockDevice custom resource for each device. Note that NDM will filter out some of the devices like loopback device and so forth as configured in the NDM ConfigMap. `kubectl get bd -n openebs`\n- NDM creates a special type of devices called sparse devices depending on the `SPARSE_FILE_COUNT` and `SPARSE_FILE_SIZE` passed to the NDM Daemon. These devices are used in cases where nodes do not have any additional devices attached to the node and users would like to run their applications by carving out some spaces from the OS disk. The creation of sparse devices is disabled by default from OpenEBS 1.3.\n- Users or Operators like cStor Operator, Local PV provisioner interact with NDM by creating a BlockDeviceClaim CR. The BlockDeviceClaim will have properties like nodeName, required Capacity etc., The NDM operator will match these properties with the available BlockDevices and associate the one that matches all the requirements to BlockDeviceClaim.\n\n---\n\n### NDM Known Issues / Future Development Items\n\n- BlockDevices are not created for Partitions and LVM devices. If you need to use them, you have to manually create BlockDevice CR. The steps are mentioned in this [blog](https://blog.mayadata.io/openebs/creating-manual-blockdevice).\n\nOK. Let us get started with some common issues reported and how to troubleshoot them.\n\n---\n\n#### Scenario #1\n\n**BlockDevice CR is not created for a device available on my node.**\n\nI have some disks attached to the node. Installed OpenEBS, but blockdevice resources are not created for the devices.\n\n**Symptom:** I have some disks attached to the node. Installed OpenEBS, but blockdevice resources are not created for the devices.\n\n**Troubleshooting:**\n\n1. Check `lsblk` output of the node\n2. Get the NDM config map.\n3. Check if the mount point of the disk is excluded in the filter configurations in configmap.\n4. From lsblk output check if the blockdevice you want to use is an LVM/software raid/ partition/LUKS filesystem. NDM currently does not support these types.\n5. If none of the above works, the logs of NDM daemonset can be checked. It will have information of disk being detected, and at what point the disk was excluded from blockdevice creation, (like `excluded by path-filter`)\n\n ![blockdevice creation](https://cdn-images-1.medium.com/max/800/0*q8rBQFw284gRYqjg)\n\n**Resolution:** Update the filter configuration in configmap and restart the NDM DaemonSet pod. This will create the blockdevices.\n\n---\n\n#### Scenario #2\n\n**After node reboot, one blockdevice became inactive and another blockdevice was created.**\n\n**Symptom:** When a node in the cluster is rebooted, A blockdevice resource on that node was marked as inactive and a new resource was created. The new blockdevice also has the same details as the old one.\n\n**Troubleshooting:**\n\n1. Check `lsblk` output of the node\n2. Get the yaml of both blockdevices and compare them.\n3. Check to see `spec.Path` is different in both outputs.\n4. If yes, then the new blockdevice resource was created because the path changed\n5. Check if `kubernetes.io/hostname` is different, if yes, then the blockdevice was created because the hostname of the node changed.\n\n**Resolution:** If using cStor, the newly generated BD can be added in both SPC and CSP instead of the old BD resource. Thus the storage engine will claim the new BD resource and start using it.\n\n**Root Cause:** Whenever the NDM deamonset pods shutdown, all the devices on that node will be marked into an Unknown state. When the pod comes backup, all the devices on that node are marked as inactive, and then individual devices are processed for their statuses.\n\nNDM uses an md5 sum of WWN+Model+Serial of the disk to create its unique name. If none of these fields are available then NDM uses device path and hostname to create the blockdevice. There are chances that the device path/hostname has changed after reboot. If the path/hostname changes a new blockdevice resource will be created, and the old one will still be in the inactive state.\n\n---\n\n#### Scenario #3\n\n**BlockDevices are created for already used disks in which OS is installed**\n\n**Symptom:** NDM created blockdevice resources for disks which are already used for OS partitions. By default NDM excludes the blockdevices that are mounted at `/, /boot, /etc/hosts`. If these mount points are on an LVM or SoftRaid, NDM will not be able to identify that.\n\n**Resolution:** Support for LVM and software RAID is in the design phase. Once it is supported the issue will be resolved.\n\n---\n\n#### Scenario #4\n\n**Only one Blockdevice is created, when devices are connected in multipath configuration**\n\n**Symptom:** There is a disk attached in multipath configuration to a node. i.e both sdb & sdc are the same devices. But blockdevice resource is created only for sdc.\n\n**Resolution:** Support for detecting disks in multipath configuration and attaching the same disk to multiple nodes will be available in the future versions of NDM\n\n**Root Cause:** NDM generates the UID for disk identification using the disk details like WWN, Serial, etc that are fetched from the disk. In case of a disk attached in multipath configuration, the details from both sdb and sdc will be the same. Therefore, NDM will first create a blockdevice for sdb, and then moves on to create for `sdc`. But at this stage, it will find that a blockdevice with that UID already exists and will update the blockdevice information with the new path `sdc`. This results in a blockdevice existing only for sdc.\n\n---\n\n#### Scenario #5\n\n**Only single BlockDevice resource is created in a multi-node Kubernetes cluster on GKE.**\n\n**Symptom:** On a multinode kubernetes cluster in GKE, with an external GPD attached to each node. NDM is creating only one blockdevice resource, instead of one blockdevice resource per node.\n\n**Troubleshooting:**\n\n1. Was the GPD added using the gcloud CLI or google cloud console web UI?\n2. If the disk was added using gcloud CLI, check whether the ` — device-name` flag was specified during attaching the disk.\n\n**Resolution:** The command to add disk using gcloud CLI should be\n\n gcloud compute instances attach-disk --disk=disk-1 **--device-name=disk-1**\n\n**Root Cause:** gcloud CLI uses the value provided in the `device-name` flag as the serial number of the GPD when it is attached to the node. If it is left blank, Google will assign a default serial number that is unique only to the node. When multiple nodes are present, and NDM generates the UID for the blockdevice, the disks on both nodes will have the same serial number and thus the same UID.\n\nNDM from one node will create the blockdevice resource and when the other NDM daemon tries to create the resource, it finds that a resource already exists and just updates the resource.\n\nThis blog was originally published on Jan 7th, 2020 on [MayaData blog](https://blog.mayadata.io/openebs/openebs-node-device-management-ndm-troubleshooting-tips).\n","notHasFeatureImage":false,"slug":"openebs-node-device-management-(ndm) — troubleshooting-tips"},{"id":39,"title":"cStor Pool Operations via CSPC in OpenEBS","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"03-01-2020","tags":["Container Attached Storage","Cspc","Kubernetes","OpenEBS","Operations"],"excerpt":"CStor Data Engine is popular for workloads needing efficient capacity management, replicas, point in time snapshots, incremental backups, etc. The tutorial will provision striped cStor pools and perform operations on them.","content":"\n**An enhanced schema for OpenEBS cStor Pool Management**\n\nCStor Data Engine is popular for workloads needing efficient capacity management, replicas, point in time snapshots, incremental backups, etc. Since achieving Beta last year, cStor Data Engine has been powering many Stateful Applications, including a variety of databases.\n\nWhile users love cStor for its data management capabilities, they have been providing feedback that it is not as easy to use as Jiva. We have started working on user feedback starting with OpenEBS 1.2 release and incrementally enhancing the cStor functionality.\n\nAs cStor is already running in production, the new changes are being introduced via a set of new cStor Custom Resources, which will get into in a bit. The users can continue to use the current schema, as we stabilize the new design and support a seamless migration. You can help us improve by providing feedback on the new design [here](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/cstor-operator) or by raising [issues](https://github.com/openebs/openebs/issues).\n\nTo give a high-level overview of the new design, the following will be changed:\n\n- A new component/deployment called cspc-operator will be deployed for managing the new custom resources.\n- CSPC — cStor Pool Cluster will be replacing the SPC. The spec of the CSPC has been modified to provide cleaner abstractions for managing blockdevices on each node.\n- CSPI — cStor Pool Instance will be replacing CSP.\n\nThe new schema will only support manual cStor Pools creation as we learned that supporting both manual and auto using a single customer CR can lead to more confusion. In future releases, a new schema will be introduced to support the automatic creation of the cStor Pools.\n\nCSPC can be used to provision cStor pool as well as carry out day 2 pool operations such as: Following storage day 2 operations are supported via the CSPC schema:\n\n- Pool Expansion ( Supported in OpenEBS version >=1.2, alpha )\n- Pool Deletion ( Supported in OpenEBS version >=1.2, alpha )\n- Pool Scale Up ( Supported in OpenEBS version >=1.2, alpha )\n- Block Device Replacement ( Supported in OpenEBS version >=1.5, alpha)\n\nLet us go through a short hands-on tutorial to understand these.\n\nThe tutorial will provision striped cStor pools and perform operations on them (this can be done for other cStor raid groups too and I will post that in a separate blog).\n\n#### Prerequisite Steps\n\n- Kubernetes cluster of version >= 1.14 with 3 worker nodes.\n- Attach at least 2 disks to each of the nodes to follow up with the tutorial. I am using GKE and steps to create and attach a disk to a node are given in the following link :\n[https://cloud.google.com/sdk/gcloud/reference/compute/disks/create\n](https://cloud.google.com/sdk/gcloud/reference/compute/disks/create)[https://cloud.google.com/sdk/gcloud/reference/compute/instances/attach-disk\n](https://cloud.google.com/sdk/gcloud/reference/compute/instances/attach-disk)If you are using other providers, check with their reference manuals on how to attach a disk. Also, feel free to reach out on the OpenEBS slack channel if you need any assistance.\n- I have used the following script to create 6 disks.\n *{ for i in {1..6}; do gcloud compute disks create demo-disk-$i — zone=us-central1-a — size=100GB; done; }*\n- Now, I will attach 2 disks to each of the nodes. I have used the following commands to attach.\n\n *for i in {1..2}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41–2ldq — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n *for i in {3..4}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41-bb69 — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n *for i in {5..6}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41-hrmr — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n- Install OpenEBS 1.5. Run following to install: (Note that block device replacement support starts from OpenEBS version 1.5. There will be a separate blog post to describe that.)\n ```\n kubectl apply -f [https://openebs.github.io/charts/openebs-operator-1.5.0.yaml](https://openebs.github.io/charts/openebs-operator-1.6.0.yaml)\n ```\n- Install CStor-Operator. Run following to install:\n ```\n kubectl apply -f [https://raw.githubusercontent.com/openebs/openebs/master/k8s/cstor-operator.yaml](https://raw.githubusercontent.com/openebs/openebs/master/k8s/cstor-operator.yaml)\n ```\n\n#### Pool Provisioning\n\nYou need to specify cStor pool intent in a CSPC YAML to provision cStor pools on nodes. I am going to provision 3 stripe cStor pools. Let us prepare a CSPC YAML now.\n\nFollowing command list all block devices which represent your attached disks. I am going to pick 1 block device from each node to form a CSPC YAML.\n\n $ kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n\n blockdevice-474d20376a541a8fb372d44f5bc361ea gke-cstor-demo-default-pool-3385ab41-hrmr 107374182400 Unclaimed Active 34s\n\n blockdevice-9773ccb731e4e3e10c2838411f5f8b2a gke-cstor-demo-default-pool-3385ab41-bb69 107374182400 Unclaimed Active 37s\n\n blockdevice-9c8df120e17379bfd1fe5c3ce9aa8185 gke-cstor-demo-default-pool-3385ab41-bb69 107374182400 Unclaimed Active 37s\n\n blockdevice-ada8ef910929513c1ad650c08fbe3f36 gke-cstor-demo-default-pool-3385ab41-2ldq 107374182400 Unclaimed Active 34s\n\n blockdevice-d2d59218ed78560b206143ab0641470c gke-cstor-demo-default-pool-3385ab41-hrmr 107374182400 Unclaimed Active 34s\n\n blockdevice-f6408e135943e1ee45171034655a8b88 gke-cstor-demo-default-pool-3385ab41-2ldq 107374182400 Unclaimed Active 34s\n\nMy CSPC YAML is on the following link (you can copy/download and make changes accordingly).\n\n[https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml](https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml)\n\nSave the above file with your changes and apply the above YAML to provision cStor pools.\n\n $ kubectl apply -f https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml\n cstorpoolcluster.openebs.io/cspc-stripe created\n\n $ kubectl get cspi -n openebs\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-8vtx gke-cstor-demo-default-pool-3385ab41-2ldq 69.5K 99.5G 99.5G ONLINE 99s\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 69.5K 99.5G 99.5G ONLINE 99s\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 69.5K 99.5G 99.5G ONLINE 99s\n\nIf you want to deploy a workload to use the cStor pool, go through the following cStor CSI driver link:\n[https://github.com/openebs/cstor-csi/blob/master/README.md](https://github.com/openebs/cstor-csi/blob/master/README.md)\n\nIn the next section, we will do pool expansion.\n\n#### Pool Expansion\n\nLet us expand one cStor stripe pool on a node by editing the CSPC cspc-stripe.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n\nSimply, add one more block device as follows. Make sure you do not put a block device that is a part of any other CSPC. I have added block device *blockdevice-d2d59218ed78560b206143ab0641470c* in my case.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n\n ...\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: gke-cstor-demo-default-pool-3385ab41-hrmr\n poolConfig:\n cacheFile: \"\"\n compression: \"off\"\n defaultRaidGroupType: stripe\n overProvisioning: false\n raidGroups:\n - blockDevices:\n - blockDeviceName: blockdevice-474d20376a541a8fb372d44f5bc361ea\n capacity: \"\"\n devLink: \"\"\n # Block device added. This must be attached to node\n # gke-cstor-demo-default-pool-3385ab41-hrmr and should not be used by any other CSPC.\n - blockDeviceName: blockdevice-d2d59218ed78560b206143ab0641470c\n\n isReadCache: false\n isSpare: false\n isWriteCache: false\n type: stripe\n ...\n\nSave the above changes.\n\n $ kubectl get cspi -n openebs\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-8vtx gke-cstor-demo-default-pool-3385ab41-2ldq 268K 99.5G 99.5G ONLINE 17m\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 292K 99.5G 99.5G ONLINE 17m\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 232K 199G 199G ONLINE 17m\n\nYou can see that the pool cspc-stripe-x9pw got expanded. (Re-run the get cspi command if the bigger size is not shown instantaneously). Similarly, you can expand the other stripe cStor pools of the CSPC.\n\n#### Pool Deletion\n\nTo delete a cStor pool from node simple remove the node configuration from the CSPC.\n\nLet us delete one pool from CSPC cspc-stripe.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n I have removed following entire config from the CSPC to delete pool on host gke-cstor-demo-default-pool-3385ab41-2ldq.\n ...\n - nodeSelector:\n kubernetes.io/hostname: gke-cstor-demo-default-pool-3385ab41-2ldq\n poolConfig:\n cacheFile: \"\"\n compression: \"off\"\n defaultRaidGroupType: stripe\n overProvisioning: false\n raidGroups:\n - blockDevices:\n - blockDeviceName: blockdevice-ada8ef910929513c1ad650c08fbe3f36\n capacity: \"\"\n devLink: \"\"\n\n isReadCache: false\n isSpare: false\n isWriteCache: false\n type: stripe\n ...\n\nAfter you are done removing, save it and you will see that pool on that node has been deleted.\n\n $ kubectl get cspi -n openebs\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 335K 99.5G 99.5G ONLINE 25m\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 372K 199G 199G ONLINE 25m\n\nYou can see that only two stripe pools are present.\n\n#### Pool Scale Up\n\nWe can even create a pool on other nodes by adding node config to the CSPC YAML. It is just the reverse of pool deletion.\n\nLet us try to add the same node config that we removed to again create the pool on the node. Make sure that while copy, pasting and editing indentation of YAML is not disturbed else error will be thrown and YAML will not be persisted.\n\nAdd the removed config from the pool deletion section to the CSPC.\n\n $ kubectl get cspi -n openebs\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 318K 99.5G 99.5G ONLINE 30m\n\n cspc-stripe-twvv gke-cstor-demo-default-pool-3385ab41-2ldq 50K 99.5G 99.5G ONLINE 9s\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 380K 199G 199G ONLINE 30m\n\nYou can see that a new pool cspc-stripe-twvv has come up online.\n\nYou can also delete the CStorPoolCluster too. If you do so, all the CStorPoolInstances associated with it will get deleted.\n\nThe command is :\n ```\n $ kubectl delete cspc \n ```\nNOTES:\n\n- Whenever a block device is used for pool creation or expansion a blockdeviceclaim CR is created and the block device will show a Claimed status.\n The following are the commands to visualize this.\n ```\n kubectl get blockdevice -n openebs \n kubectl get blockdeviceclaim -n openebs\n ```\n- Whenever a pool is deleted for a CSPC by removing the node config, the associated block-device is not ‘Unclaimed’ and if the same block device needs to be used in another CSPC, the associated blockdeviceclaim needs to be cleared manually. Although, the block-device can be again used for the same CSPC. \n\nTo unclaim a block device claim, below are the steps.\n\n\n $ kubectl get bdc -n openebs\n NAME BLOCKDEVICENAME PHASE AGE\n\n bdc-cstor-83b8e958-d978-11e9-b0e6-42010a800189 blockdevice-9773ccb731e4e3e10c2838411f5f8b2a Bound 32m\n\n bdc-cstor-8581dac0-d978-11e9-b0e6-42010a800189 blockdevice-ada8ef910929513c1ad650c08fbe3f36 Bound 32m\n\n bdc-cstor-85bddd0e-d978-11e9-b0e6-42010a800189 blockdevice-474d20376a541a8fb372d44f5bc361ea Bound 32m\n\n bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 blockdevice-d2d59218ed78560b206143ab0641470c Bound 15m\n\nLet us say, we want to unclaim block device blockdevice-d2d59218ed78560b206143ab0641470c, then we need to delete the associated blockdeviceclaim i.e. bdc-cstor-85c0fd2a-d978–11e9-b0e6–42010a800189.\n\n $ kubectl edit bdc bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 -n openebs\n Remove the finalizer \"cstorpoolcluster.openebs.io/finalizer\" by editing the bdc from above command.\n After that, execute following\n\n $ kubectl delete bdc bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 -n openebs\n\nNow the device will get unclaimed. Please note the following:\nBDC CRs should be deleted only when their association with CSPC has been removed. Otherwise, data corruption can happen.\n\n#### Block Device Replacement\n\nThe CSPC operator in OpenEBS (≥1.5 version ) enables you to do the replacement of the block devices in case it has gone bad. I will follow up with another blog post that will cover this block device replacement scenario.\n\nHope that the tutorial helps in understanding the pool operations steps.\n\nIf you have any questions or face any problems, feel free to reach out to me on OpenEBS slack channel.\n\nThank You!\n\nThis blog was originally published on Dec 13, 2019 on [MayaData blog](https://blog.mayadata.io/openebs/cstor-pool-operations-via-cspc-in-openebs).\n","notHasFeatureImage":false,"slug":"cstor-pool-operations-via-cspc-in-openebs"},{"id":40,"title":"OpenEBS Dynamic Volume Provisioning on ZFS","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"13-12-2019","tags":["CNCF","Kubernetes","Persistent Volume","ZFS","OpenEBS"],"excerpt":"OpenEBS’ ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. This blog will demonstrate how to deploy a Percona application on the ZFS storage system with OpenEBS.","content":"\nOpenEBS’ ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. This blog will demonstrate how to deploy a Percona application on the ZFS storage system with OpenEBS.\n\nUsing a ZFS Local PV has the following advantages — as opposed to Kubernetes native Local PV backed by direct attached devices:\n\n- Sharing of the devices among multiple application pods.\n- Enforcing quota on the volumes, making sure the pods don’t consume more than the capacity allocated to them.\n- Ability to take snapshots of the Local PV\n- Ability to sustain the disk failures — using the ZPOOL RAID functionality\n- Ability to use data services like compression and encryption.\n\nIn this post, I will demonstrate how we can use ZFS Local PV for deploying a Percona application.\n\n**Setup**\n\nWe will be using GKE with Kubernetes 1.14+ version with Ubuntu 18.4 installed on each node. We have to set up the node with ZFS utility once the cluster is up and running. We need to install [zfsutils-linux](https://packages.ubuntu.com/bionic/zfsutils-linux) on each node and use [ZFS commands](https://www.thegeekdiary.com/solaris-zfs-command-line-reference-cheat-sheet/) to set up a storage pool.\n\n $ apt-get install -y zfsutils-linux\n\nCreate and attach the disk (if not already attached) to the nodes for setting up the ZPOOL:\n\n $ gcloud compute disks create --size --type pd-standard --zone us-central1-a\n\n $ gcloud compute instances attach-disk --disk --zone us-central1-a\n\nCreate the zpool on each node using the attached disks (sdb and sdc):\n\n $ zpool create zfspv-pool mirror /dev/sdb /dev/sdc\n\nHere we are creating a mirrored ZPOOL, we can create any kind of pool as per our requirement (raidz, striped or mirror).\n\nCheck the zpool list:\n\n $ zfs list\n NAME USED AVAIL REFER MOUNTPOINT\n\n zfspv-pool 644K 96.4G 176K /zfspv-pool\n\nInstall OpenEBS ZFS driver :\n\n $ kubectl apply -f [https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml](https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml)\n\nCheck that the driver is installed:\n\n $ kubectl get pods -n kube-system -l role=openebs-zfs\n\n NAME READY STATUS RESTARTS AGE\n\n openebs-zfs-controller-0 4/4 Running 0 5h28m\n\n openebs-zfs-node-4d94n 2/2 Running 0 5h28m\n\n openebs-zfs-node-gssh8 2/2 Running 0 5h28m\n\n openebs-zfs-node-twmx8 2/2 Running 0 5h28m\n\n**Create The Storage Class:**\n\n $ cat sc.yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: percona-sc\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nThe storage class has a *poolname* parameter, which means that any volume provisioned using this storage class will be provisioned in this pool (zfspv-pool here). The provisioner *zfs.csi.openebs.io* is the provisioner name for the ZFS driver. You can change the poolname to the name of the ZPOOL which you have created for the provisioning. Apply the YAML to create the storage class:\n\n $ kubectl apply -f sc.yaml\n\n storageclass.storage.k8s.io/percona-sc created\n\n**Create The PVC:**\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: percona-pvc\n spec:\n storageClassName: percona-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nCreate the PVC using the storage class created for the ZFS driver. You can request for the storage space via storage parameter as shown in above PVC for putting a request for 4Gi storage. Apply the YAML to create the PVC request.\n\n $ kubectl apply -f pvc.yaml\n\n persistentvolumeclaim/percona-pvc\n\nCheck the PVC\n\n $ kubectl get pvc\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n\n percona-pvc Bound pvc-ecdb16e2-e03a-11e9-b418–42010a80006b 4Gi RWO percona-sc 5m39s\n\nHere, we can see that the PVC has been created and bound also as well.\n\n**Percona YAML:**\n\n apiVersion: v1\n kind: ConfigMap\n metadata:\n annotations:\n name: sqltest\n namespace: default\n data:\n sql-test.sh: |\n #!/bin/bash\n\n DB_PREFIX=\"Inventory\"\n DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`\n DB_NAME=\"${DB_PREFIX}_${DB_SUFFIX}\"\n\n echo -e \"nWaiting for mysql server to start accepting connections..\"\n retries=10;wait_retry=30\n for i in `seq 1 $retries`; do\n mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1\n rc=$?\n [ $rc -eq 0 ] && break\n sleep $wait_retry\n done\n\n if [ $rc -ne 0 ];\n then\n echo -e \"nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exitingn\"\n exit 1\n fi\n mysql -uroot -pk8sDem0 -e \"CREATE DATABASE $DB_NAME;\"\n mysql -uroot -pk8sDem0 -e \"CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"INSERT INTO Hardware (id, name, owner, description) values (1, \"dellserver\", \"basavaraj\", \"controller\");\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"DROP DATABASE $DB_NAME;\"\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: percona\n labels:\n name: percona\n spec:\n replicas: 1\n selector:\n matchLabels:\n name: percona\n template:\n metadata:\n labels:\n name: percona\n spec:\n containers:\n - resources:\n name: percona\n image: openebs/tests-custom-percona:latest\n imagePullPolicy: IfNotPresent\n args:\n - \"--ignore-db-dir\"\n - \"lost+found\"\n env:\n - name: MYSQL_ROOT_PASSWORD\n value: k8sDem0\n ports:\n - containerPort: 3306\n name: percona\n volumeMounts:\n - mountPath: /var/lib/mysql\n name: demo-vol1\n - mountPath: /sql-test.sh\n subPath: sql-test.sh\n name: sqltest-configmap\n livenessProbe:\n exec:\n command: [\"bash\", \"sql-test.sh\"]\n initialDelaySeconds: 30\n periodSeconds: 1\n timeoutSeconds: 10\n volumes:\n - name: demo-vol1\n persistentVolumeClaim:\n claimName: percona-pvc\n - name: sqltest-configmap\n configMap:\n name: sqltest\n\n ---\n apiVersion: v1\n kind: Service\n metadata:\n name: percona-mysql\n labels:\n name: percona-mysql\n spec:\n ports:\n - port: 3306\n targetPort: 3306\n selector:\n name: percona\n\nApply the configuration:\n\n $ kubectl apply -f percona.yaml\n\n configmap/sqltest created\n deployment.apps/percona created\n service/percona-mysql created\n\nCheck the status of the Pod.\n\n $ kubectl get po\n\n NAME READY STATUS RESTARTS AGE\n\n percona-7456dc6f7b-nnfcz 1/1 Running 0 67s\n\nWe can go the node where percona pod is scheduled and see that a volume has been created in the pool zfspv-pool using the ZFS list command,:-\n\n $ zfs list\n\n NAME USED AVAIL REFER MOUNTPOINT\n\n zfspv-pool 115M 96.3G 176K /zfspv-pool\n\n zfspv-pool/pvc-ecdb16e2-e03a-11e9-b418–42010a80006b 102M 96.3G 102M -\n\n### Summary\n\nAs demonstrated in this blog, OpenEBS makes it easy for the Kubernetes applications to take advantage of all the technical features provided by ZFS storage.\n\n### Important links\n\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n","notHasFeatureImage":false,"slug":"openebs-dynamic-volume-provisioning-on-zfs"},{"id":41,"title":"ECK & OpenEBS — Data Ops Streamlines Deployment","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"23-08-2019","tags":["Eck","Elasticsearch","Kubernetes","LocalPV","OpenEBS"],"excerpt":"Using OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch ECK scaling and resilience while finally delivering a completely declaratively-managed application stack.","content":"\n### TL;DR\n\nUsing OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch ECK scaling and resilience while finally delivering a completely declaratively-managed application stack. Let’s review how.\n\n**Good News: The recently shipped** [**Elastic Cloud on Kubernetes (ECK)**](https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond) delivers Elasticsearch clusters as native, distributed Kubernetes Operators. Elasticsearch is a distributed, open source search and analytics engine for all types of data. Widely used, Elasticsearch is storage-intensive because it builds an inverted index of collections of JSON objects that are related to each other to allow very fast full-text searches. The result is a simplified deployment of ElasticSearch for the Kubernetes admins or SREs as well as a simplified developer experience.\n\n**Bad News: ElasticSearch uses fast local storage but it does not address storage provisioning and management.** ECK use of the [static provisioner for Local](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) PV requires administrators to manually format, mount, and configure disks on Kubernetes nodes. This is a PITA, as they say.\n\n**Good News: OpenEBS removes the burden of Storage Operations for ECK Deployments.** With OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch scaling and resilience.\n\n### Vanilla K8s (PITA)\n\nTypically, ElasticSearch is deployed one of two ways:\n\n- **Dedicated mode:** Elastic pods are using LocalPV which are real disks, and they need to be dynamically provisioned as the pods’ scale.\n- **Shared mode:** Elastic pods are using LocalPVs from shared storage for better capacity economics.\n\n Here is the detail on K8s static provisioners from [K8s documentation](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner)\n\n *Note that the local storage provisioner is different from most provisioners and does not support dynamic provisioning. Instead, it requires that administrators preconfigure the local volumes on each node and if volumes are supposed to be*\n\n- *Filesystem volumeMode (default) PVs — mount them under discovery directories.*\n- *Block volumeMode PVs — create a symbolic link under discovery directory to the block device on the node.*\n\n *The provisioner will manage the volumes under the discovery directories by creating and cleaning up PersistentVolumes for each volume.*\n\n *This means a Kubernetes administrator must manually manage storage operations outside of the ECK operator itself. Specifically,*\n\n *– Capacity resize of underlying volumes.*\n\n *– Capacity management of shared storage when deployed in shared mode.*\n\n *– Shifting of some of the data volumes from one node to another automatically in case a node has to be cordoned or drained.*\n\n *– Move data to other Kubernetes clusters.*\n\n**Note: insert administrator sweat equity here^.**\n\n### OpenEBS LocalPV + Data Ops (Good)\n\nAn OpenEBS storage cluster, itself Kubernetes-native, simplifies and automates storage provisioning and management operations either on your data center or in the cloud (or spanning both!). OpenEBS provisioners use OpenEBS disk pool operators and built-in data management capabilities to dynamically provision LocalPVs in a host path or in a disk mode to ECK pods.\n\n![Multiple elastic data nodes sharing a pool of disks for thim provisioning effect](https://cdn-images-1.medium.com/max/800/1*PHw4zrcvJF_w-VcTI90RbA.png)\n\n### K8s Advantage: Declarative Data Plane\n\nOpenEBS uses a Declarative Data Plane to manage storage operations which aligns architecturally with Kubernetes Operators generally, and specifically with the ECK operator. Storage is typically the last “architectural mile” of Kubernetes deployments. Storage has gravity for applications that tends to tie Kubernetes clusters to the storage infrastructure beneath it. The OpenEBS Declarative Data Plane removes that last architectural mile by providing a completely Kubernetes-native software-defined storage infrastructure that spans on-premise and cloud resources and lets administrators manage Kubernetes application gravity consistently across all sites.\n\n**OpenEBS-managed storage means that the end-to-end operations of the entire Kubernetes application stack is finally managed, top to bottom, in a completely declarative way.**\n\n### Configuring a Dynamic localPV for ECK\n\nThe StorageClass spec for [OpenEBS LocalPV](/docs/concepts/localpv) for automatically choosing an available disk on the node and mounting that disk with ext4 volume would look like the following:\n\n cat <> ~/.ssh/authorized_keys' \n eval 'ssh-agent'\n \n - name: Copying the SSH key into compute nodes\n shell: |\n ssh -o StrictHostKeyChecking=no root@{{item}} 'echo >> ~/.ssh/authorized_keys'\n eval 'ssh-agent' \n with_items: \"\"\n \n - name: SSH from master to each Nodes \n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@{{item}} ls'\n with_lines: cat ./ip.csv\n \n - name: Generating random number\n shell: date +%s\n register: rand_num\n \n - name: Setting up the master hostname \n shell: |\n ssh -o StrictHostKeyChecking=no root@ 'echo master-. > /etc/hostname'\n ssh -o StrictHostKeyChecking=no root@ 'systemctl start systemd-hostnamed'\n \n - name: Setting up the compute nodes hostname\n shell: |\n ssh -o StrictHostKeyChecking=no root@ 'echo node-. > /etc/hostname'\n ssh -o StrictHostKeyChecking=no root@ 'systemctl start systemd-hostnamed' \n with_together: \n - [ '1', '2', '3' ]\n - \"\" \n \n #The VMs are already subscribed with some credentials. Need to unsubscribe the VMs & subscribe it again with new credentials.\n - name: Unsubscribing the nodes\n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager unregister'\n ignore_errors: true\n with_lines: cat ./ip.csv \n \n - name: Subscribing the nodes\n shell: |\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager register --username= --password='\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager attach --auto'\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager refresh'\n with_lines: cat ./ip.csv\n \n - name: Getting the pool-id\n shell: ssh -o StrictHostKeyChecking=no root@ 'subscription-manager list --available --matches '*OpenShift*' | grep \"Pool ID\" | awk '\\''NR == 1'\\'' | awk '\\''{print $3}'\\'''\n register: pool_id\n \n - name: Attaching pool to each nodes\n shell: | \n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager attach --pool='\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager repos --enable=\"rhel-7-server-rpms\" --enable=\"rhel-7-server-extras-rpms\" --enable=\"rhel-7-server-ose-3.10-rpms\" --enable=\"rhel-7-server-ansible-2.4-rpms\"'\n with_lines: cat ./ip.csv\n \n - name: Joining the nodes to the DNS Server\n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'echo | realm join --user=Administrator '\n with_lines: cat ./ip.csv\n \n - name: SSH from master to master using DNS\n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@master-. ls' \n \n - name: SSH from master to compute nodes using DNS\n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@node{{item}}-. ls'\n with_items: \n - 1\n - 2\n - 3 \n \n - name: Replacing master ip in inventory\n replace:\n path: ./inventory.yml\n regexp: \"master_ip\"\n replace: \"\" \n \n - name: Replace compute node ip in inventory\n replace:\n path: ./inventory.yml\n regexp: \"\"\n replace: \"\"\n with_together: \n - [ 'node1_ip', 'node2_ip', 'node3_ip' ]\n - \"\" \n \n - name: Replacing master DNS in inventory\n replace:\n path: ./inventory.yml\n regexp: \"master_dns\"\n replace: \"master-.\"\n \n - name: Replacing nodes DNS in inventory\n replace:\n path: ./inventory.yml\n regexp: \"\"\n replace: \"\"\n with_together:\n - [ 'node1_dns', 'node2_dns', 'node3_dns' ]\n - [ 'node1-.', 'node2-.', 'node3-.' ] \n \n - name: Copying inventory into master\n shell: scp -o StrictHostKeyChecking=no inventory.yml root@:/root/openshift-ansible/inventory/\n \n - name: Checking out to release branch-3.10\n shell: ssh -o StrictHostKeyChecking=no root@ 'cd /root/openshift-ansible && git checkout release-3.10'\n \n - name: Running Openshift pre-requisites\n shell: ssh -o StrictHostKeyChecking=no root@ 'ansible-playbook -i /root/openshift-ansible/inventory/inventory.yml /root/openshift-ansible/playbooks/prerequisites.yml -vv'\n \n - name: Deploying openshift cluster\n shell: ssh -o StrictHostKeyChecking=no root@ 'ansible-playbook -i /root/openshift-ansible/inventory/inventory.yml /root/openshift-ansible/playbooks/deploy_cluster.yml -vv'\n \n - name: Disabling selinux on each nodes \n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'setenforce 0'\n with_lines: cat ./ip.csv \n \n - name: Copying bash file in master & Executing\n shell: | \n scp -o StrictHostKeyChecking=no post_install_setting.sh root@:/root/ \n ssh -o StrictHostKeyChecking=no root@ 'bash post_install_setting.sh && rm post_install_setting.sh'\n\n### App deployers\n\nLitmus job for deploying NuoDB EE using OpenEBS volumes for database requirements.\n\nPrerequisites for running this litmus job is to have NuoDB Enterprise edition subscription and have docker images available in the cluster.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/OpenShift/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/OpenShift/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: litmus-nuodb-\n namespace: litmus\n spec:\n template:\n metadata:\n name: litmus\n labels:\n app: nuodb-deployment\n \n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n \n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays, actionable, default\n value: default\n \n - name: PROVIDER_STORAGE_CLASS\n # Supported values: openebs-standard, local-storage, openebs-standalone\n value: openebs-cstor-sparse\n \n - name: NUODB_VERSION\n value: ee\n \n - name: APP_PVC\n value: demo-vol-claim\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n # Application label\n - name: APP_LABEL\n value: 'app=nuodb'\n \n # Use 'deprovision' for app-clean up\n - name: ACTION\n value: provision\n \n # Set THP to disable in case platform is AWS or OpenShift\n - name: THP\n value: disable\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/deployers/OpenShift/test.yml -i /etc/ansible/hosts -v; exit 0\"]\n\n### Chaos jobs — NuoDB\n\nLitmus job for inducing chaos on NuoDB application components such as Admin, Storage Manager and Transaction Engine.\n\nFor inducing various components chaos, user has to pass application component specific label as the Job Env to the litmus book.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/chaos/app_pod_failure/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/chaos/app_pod_failure/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: nuodb-app-chaos-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: nuodb-app-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: DEPLOY_TYPE\n value: statefulset\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/chaos/app_pod_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Networking\n\nLitmus job for inducing network delays between application and openebs target pod.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/experiments/chaos/openebs_target_network_delay/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/experiments/chaos/openebs_target_network_delay/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: nuodb-app-chaos-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: nuodb-app-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: DEPLOY_TYPE\n value: statefulset\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/chaos/app_pod_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Storage\n\nLitmus job for inducing OpenEBS cStor storage target kill and verify the application availability.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-target-failure-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: openebs-target-failure\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n - name: TARGET_NAMESPACE\n value: openebs\n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: DATA_PERSISTENCY\n value: \"\" \n \n # CHOS_TYPE values : target-zrepl-kill , target-kill , target-delete \n - name: CHAOS_TYPE\n value: \"target-zrepl-kill\"\n \n # TARGET_CONTAINER values: cstor-volume-mgmt , cstor-istgt\n - name: TARGET_CONTAINER\n value: \"cstor-volume-mgmt\"\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_target_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\nLitmus job for inducing OpenEBS cStor storage pool kill and verify the application availability.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-pool-failure-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: openebs-pool-failure\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n \n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: DATA_PERSISTENCY\n value: \"\" \n \n - name: CHAOS_TYPE\n value: \"pool-kill\"\n \n - name: CHAOS_ITERATIONS\n value: \"2\" \n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_pool_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Kubernetes\n\nLitmus job for inducing kubelet/docker service crash and verify impact on the application running on the node.\n\nNote: This litmus job is specific to AWS platform.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-app-svc-chaos-\n namespace: litmus \n spec:\n template:\n metadata:\n labels:\n name: openebs-app-svc-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n \n #nodeSelector:\n # kubernetes.io/hostname:\n \n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n env: \n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: OPERATOR_NAMESPACE\n value: openebs\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n # Set value to kubelet/docker \n - name: SVC_CHAOS\n value: docker\n \n - name: CHAOS_DURATION\n value: \"300\" # in seconds\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: PLATFORM\n value: \"AWS\"\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_app_svc_chaos/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n","notHasFeatureImage":false,"slug":"using-chaos-engineering-to-harden-applications-that-use-nuodb,-openebs"},{"id":44,"title":"OpenEBS community releases v1.1, maintaining a faster release cadence.","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"08-08-2019","tags":["Kubernetes","Microservices","OpenEBS","Persistent Volume","Stateful Workloads"],"excerpt":"In this blog, I will provide a quick summary of the changes that were released as part of OpenEBS version 1.1 and also share some thoughts on the evolving project management process in OpenEBS and how it is helping to maintain a faster release cadence.","content":"\nIn this blog, I will provide a quick summary of the changes that were released as part of OpenEBS version 1.1 and also share some thoughts on the evolving project management process in OpenEBS and how it is helping to maintain a faster release cadence.\n\nOpenEBS Release 1.1 has been about fixing and documenting the cross-platform usability issues reported by users and also laying the foundation for some of the long-overdue backlogs like CSI Driver, automated upgrades, day 2 operations, and others.\n\nBefore we get into the specifics of the current release, the last three OpenEBS releases have set an interesting precedent towards attaining a *monthly release cadence*.\n\nOpenEBS was built by adopting the cloud-native and microservices principles, and it is almost only natural to also reap the benefits of true DevOps product with faster releases. It is easier said than done though! After having experimented with several tools and having looked at various open-source projects including Kubernetes, we have arrived at the following process, which is helping us maintain release cadence and thereby being responsive to the user requirements.\n\n- Responsiveness — Almost all the active contributors and maintainers of the OpenEBS project are reachable and online in the OpenEBS Community Slack. OpenEBS has been credited as being one of the most responsive CNCF community projects — and thanks to the community, OpenEBS Developers are getting feedback directly from end-users. This eliminates layers of requirements for implementation and improves the feedback loop.\n- Clarity of criteria for alpha and beta — Recently we clarified that our release gates are defined by OpenEBS Litmus based GitLab pipelines that run end-to-end tests on multiple different platforms and stateful workloads. Perhaps goes without saying — however we use these pipelines to catch any regressions. What is more — a feature is marked as Beta only after it has been added to the test pipelines. For example, LocalPV as of OpenEBS 1.1 is now Beta because it is passing these tests — and also is seeing a lot of production usage as well.\n- Backlog grooming — At the start of the release, we look at the backlogs which are on [GitHub](https://github.com/openebs/openebs/issues). Items are selected based on contributor availability and balancing the development of new features, fixing existing features, updating and improving documentation, improving e2e coverage, and hardening the usage of OpenEBS on new platforms. As an example of a new platform, we have seen quite a bit of usage of especially the low footprint Jiva on ARM and are now releasing container images for built for the ARM64 architecture, making OpenEBS operational on RPi4 as well as Amazon A1 instances or Packet’s powerful ARM Compute servers. As another example, we are hardening the use of OpenEBS for Konvoy from our friends at Day2IQ — and shortly we will see Konvoy on OpenEBS.ci. As reminder OpenEBS.ci is a public way for showing that all commits to OpenEBS master are tested against a set of workloads and platforms. OpenEBS also now appears in the [OpenShift Operator Hub](https://github.com/openebs/helm-operator/blob/master/olm/README.md) and on the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/MayaData-OpenEBS-Cloud-Native-Storage/B07TFS9Q8D) as well\n- Tracking items — The list of selected items is tracked for the current release using these [Google Sheets](https://docs.google.com/spreadsheets/d/1bbphUqbxShBhgr1VHaEQUzIGMaJJacPNKc1ckNXU1QE/). It is not fancy, but it helps to get all the collaborators together and very easily provides a no-barrier objective follow-up — between release manager, leads, and reviewers. The format of the sheet is a modified version of what is used by Kubernetes sig-storage.\n- Role of core committers — As core contributors, our responsibility is to detail the design and to list the implementation tasks — including covering the integration and upgrade tests. Each granular task is updated in the above project sheet and then we ask for help from the community to fix some of these items. The designs themselves are discussed and maintained as GitHub PRs [here](https://github.com/openebs/openebs/tree/master/contribute/design).\n- Role of RC1 and RC2 — functionality must be checked into master before RC1 builds are started. Post RC1 it is mostly about corner cases, integration and upgrade tests. Only those features that can complete the upgrade testing within the RC2 timelines are considered for the current release.\n- Role of release manager — Conducts follow-ups via daily standups on pending items and mitigating the risks by seeking additional help or by pushing the feature out of the release.\n- The final two weeks — As we reach the end of a one month release cycle the focus turns to refactoring and adding more test cases while stabilizing the features rather than introducing new features. The last two weeks are also about polishing documentation and trying to reach out to users whose requests have been incorporated into the product to get some early feedback.\n- What else? I haven’t spoken about the role of beta tests or dogfooding of the releases by using OpenEBS in our own hosted services such as OpenEBS director. Perhaps I’ll dig into these in a future blog. Bookkeeping tasks that start after the release also take a lot of time. For example, OpenEBS can be deployed via different partner platforms, each of which maintains their repositories for their Helm charts. Each of these partners are evolving with new guidelines for check-ins and they tend to go at their own pace. There is definitely room for improvement here and hopefully, the way the kubernetes apps are delivered will be standardized so that such bookkeeping tasks can be reduced.\n\n*How do you run your Open Source projects? What tools do you use to improve productivity? Please drop in a comment. Would love to hear from you and improve the care and feeding of the OpenEBS community.*\n\nGetting back to OpenEBS 1.1. The major features, enhancements and bug fixes in this release include:\n\n- Upgrades! Support for the upgrade of OpenEBS storage pools and volumes through Kubernetes Job. As a user, you no longer have to download scripts to upgrade. The procedure to upgrade via Kubernetes Job is provided [here](https://github.com/openebs/openebs/tree/master/k8s/upgrades/1.0.0-1.1.0). Kubernetes Job-based upgrade is a step towards completely automating the upgrades in the upcoming releases. Would love to hear your feedback on the [proposed design](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/upgrade). Note: Upgrade job makes use of a new container image called quay.io/openebs/m-upgrade:1.1.0.\n- CSI — The CSI driver reached Alpha with initial functionality for provisioning and de-provisioning of cStor volumes. Once you have OpenEBS 1.1 installed, take the CSI driver for a spin on your development clusters using the instructions provided [here](https://github.com/openebs/csi). The addition of the CSI driver also requires a shift in the paradigm of how the configuration of the storage class parameters should be passed on to the drivers. We want to keep this seamless, please let us know if you have any inputs on what you notice as some of the nice to have as we shift towards the [CSI driver](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/csi).\n- Day 2 automation ongoing — There is a tremendous amount of work ongoing to further automate Day 2 operations of the cStor storage engine. Most of these changes did not make the current release because the nature of schema changes were larger than could be taken within the current release cycle. The feature is under active development and if you are interested in providing feedback on how this feature is shaping up, you can find the proposed design [here](https://github.com/openebs/openebs/pull/2595). Thank you to everyone that has already chipped in with ideas and feedback.\n\nPerhaps the greatest highlight of this release is an increased involvement from OpenEBS user community pitching in with GitHub Issues as well as providing contributions.\n\n![Giacomo Longo Message](https://cdn-images-1.medium.com/max/800/1*hZ7FK18EK2_PfjdCJB2OTQ.png)\n\nHere are some issues that were raised and fixed within the current release.\n\n- Fixed an issue where backup and restore of cStor volume using OpenEBS velero-plugin was failing when OpenEBS was installed through Helm. [@gridworkz](https://github.com/gridworkz)\n- Fixed an issue with NDM where the kubernetes.io/hostname for Block Devices on AWS Instances was being set as the nodeName. This was resulting in cStor Pools not being scheduled to the node as there was a mismatch between hostname and nodename in AWS instances. [@obeyler](https://github.com/obeyler)\n- Fixed an issue where NDM was seen to crash intermittently on nodes where NVMe devices are attached. There was an issue in the handling of NVMe devices with write cache supported resulting in a segfault. [Private User]\n- Added support to disable the generation of default storage configuration like StorageClasses, in case the administrators would like to run a customized OpenEBS configuration. [@nike38rus](https://github.com/nike38rus)\n- Fixed an issue where the cStor Target would fail to start when the NDM sparse path is customized. [@obeyler](https://github.com/obeyler)\n- Fixed a regression that was introduced into the cStor Sparse Pool that would cause the entire Volume Replica to be recreated upon the restart of a cStor Sparse Pool. The fix was to make sure the data is rebuilt from the peer Sparse pools instead of recreating. Test cases have been added to the e2e pipeline to catch this behavior with Sparse Pools. Note that this doesn’t impact the cStor Pools created on Block Devices. [@vishnuitta](https://github.com/vishnuitta)\n- For Jiva Volumes, created a utility that can clear the internal snapshots created during replica restart and rebuild. For long-running volumes that have gone through multiple restarts, the number of internal snapshots can hit the maximum supported value of 255, after which the Replica will fail to start. The utility to check and clear the snapshots is available [here](https://github.com/openebs/openebs/tree/master/k8s/jiva). [@rgembalik](https://github.com/rgembalik)[@amarshaw](https://github.com/amarshaw)\n- Enhanced velero-plugin to allow users to specify a backupPathPrefix for storing the volume snapshots in a custom location. This allows users to save/backup configuration and volume snapshot data under the same location rather than saving the configuration and data in different locations. [@amarshaw](https://github.com/amarshaw)\n\n***For detailed change summary, steps to upgrade from a previous version, or to get started with v1.1 please refer to:*** [***Release 1.1 Change Summary***](https://github.com/openebs/openebs/releases/tag/1.1.0)\n\nIn short, OpenEBS 1.1 shows that OpenEBS development is marching ahead faster and faster and delivering more and more features, fixes and platforms.\n\nAs always if you have any feedback or inputs regarding the OpenEBS project or project management — please reach out to me on [Slack](https://slack.openebs.io) or [GitHub](https://github.com/openebs/openebs/) or via comments here.\n\n\n\n Example yaml file:\n\n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: default\n type: hostdir\n spec:\n path: \"/mnt/disks/ssd0\"\n ---","notHasFeatureImage":false,"slug":"openebs-community-releases-v1.1,-maintaining-a-faster-release-cadence."},{"id":45,"title":"OpenEBS Project Update and whats coming in v1.0","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"16-05-2019","tags":["Cncf","Containerattachedstorage","Storage","Kubernetes","OpenEBS"],"excerpt":"My heartfelt gratitude to hundreds of contributors and thousands of users and friends of OpenEBS who have contributed to OpenEBS becoming a CNCF Sandbox Project.","content":"\nMy heartfelt gratitude to hundreds of contributors and thousands of users and friends of OpenEBS who have contributed to OpenEBS becoming a CNCF Sandbox Project.\n\nFor those of you who have just heard of OpenEBS or are curious about the current state of OpenEBS, I put together this quick blog that *Looks back to where we started and how far we have come*. (I am pretty sure the GOT fans will get this reference). I then talk about the current release 0.9 and what is to come in 1.0.\n\nThe OpenEBS vision has been pretty clear since the start: provide an Open Source Storage Solution that enables Enterprises and Solution Architects to reap the Agility benefits promised by Container Native Architectures. We decided a few years ago that delivering data agility could best be done by using the tools that deliver agility in development and operations already — cloud native architectures, including containers and Kubernetes. Specifically we decided to move Stateful Workloads and the logic that delivers storage services to protect and manage them into a microservices based architecture deployed via containers. We also set out to enable agility by fighting the lock-in of data into vendor and cloud specific storage silos that lock users into specialized systems and services that themselves require special skills to run and scale.\n\nOver the last couple of years the notion of containerizing stateful workloads and managing them via Kubernetes has become increasingly accepted. We now see a great breadth of deployments of different workloads on different flavors of Kubernetes — you can read more about how the pattern of cloud native or what we call Container Attached Storage is being adopted on the CNCF blog we helped author here: [A Year Later — Updating Container Attached Storage](https://www.cncf.io/blog/2019/05/16/a-year-later-updating-container-attached-storage/).\n\n*Quick point that you may have already gathered: OpenEBS is completely Kubernetes native and if you know how to manage your applications in Kubernetes, you already know how to use OpenEBS.*\n\nSo, OpenEBS started out to build a Storage Solution that has:\n\n- *Stable Date Engines*, can be run on any underlying Kubernetes Platform. And it has got to be multiple storage engines, so administrators can compose or pick the right one for the right workload. For instance, an OpenEBS PV for Mongo can be using a completely different data engine compared to a PV for PostgreSQL or Jenkins.\n- *Standard Specs* to manage the Data Engines, so that administrators can use the existing Kubernetes tooling infrastructure to manage storage as well. Kubernetes Custom Resources and operators are used to manage everything with regards to OpenEBS.\n- *Security* controls that enable running on platforms like OpenShift, SuSE CAAS or in Kubernetes Clusters with strict Pod Security Policy.\n- *Scalable* architecture that can deliver the required performance to the Application, without adding much of an overhead over the underlying storage medium.\n\nAnd above all, it has to be ***simple and easy to use.***\n\nIt feels great for OpenEBS to be approaching 1.0!! We are extremely humbled at the amount of love in terms of both support and scrutiny that we have received over the last couple of years as we went through with building each block by block completely in the Open Source. OpenEBS has clearly established itself as the most simple to use and cost effective storage solution that is available out there to use with Kubernetes — and yet we know there is much more to do, especially as there seems to be a risk of higher level commands such as those covering data mobility being pulled back from Kubernetes into their own systems by proprietary vendors. More on that in a later blog and/or set of GitHub issues :) .\n\nIn the following sections, I go through in a bit more detail the current state of the main aspects of OpenEBS — Stability, Standard Spec (aka Storage Policies), Security Controls and Scalable architecture; followed by a quick summary of the current release (0.9) and what we are working on currently for v1.0.\n\n**Data Engines:** The data engines are the containers responsible for interfacing with the underlying storage devices or cloud volumes such as host filesystem, rotational drives, SSDs and NVMe devices. The data engines provide volumes with required capabilities like high availability, snapshots, clones, etc. Depending on the capabilities required, users can select the right data engine like cStor ( a CoW based) or Jiva or even Local PVs for a given volume.\n\nWe define a data engine as stable if it meets the following criteria:\n\n- Resilient against node, network or storage device errors. No data loss!\n- Ease of management ( including Day 2 Operations).\n- Users are running the data engine in production for more than 6 months and have gone through two or more OpenEBS version upgrades.\n\nThe current state of the 3 data engines supported by OpenEBS are as follows:\n\n- **Jiva (stable)** — The first and basic data engine that was supported by OpenEBS and has been deployed the longest in production by users. We have fixed several issues that came up with regards to cluster upgrades, node migrations, storage expansion and so forth. Ideal for cases where only replication of data is required. For backup and restore, Velero/Restic is used. Very easy to use, lightweight.\n- **cStor (beta)** — The most feature rich data engine that has the support for extremely efficient snapshots and clones. Highly recommended for cases where the nodes have storage devices attached. The current release contains Velero OpenEBS plugins that help with optimizing the backup/restore process. While already used in production by some of our users, we consider this as beta — until all day 2 operations are easily performed.\n- **OpenEBS Local PV (alpha)** — An extension to Kubernetes Local PV, with the plan to ease the management of disks by using the OpenEBS Node Storage Device Manager (NDM). The current OpenEBS 0.9 release contains the initial version of the OpenEBS Local PV Dynamic Provisioner. While we have tested OpenEBS Local PV extensively however per the criteria above it remains alpha as the newest engine. From an IO engine perspective, it is as stable as it can get. :-)\n\nAdditional details and how each of the Data engines operate are provided in this [Presentation](https://docs.google.com/presentation/d/1mjOkAQppyd23sw7PIryxu5kSrex352bT6bINzw6mUFY/edit?usp=sharing)\n\n![OpenEBS cStor Volume](/images/blog/cas-example-openebs-cstor-volume.png)\n\n**Standard Specifications or API:** Standardization is achieved by architecting OpenEBS as a set of microservices using Kubernetes Custom Resources and Operator patterns. The same set of tooling used to manage the applications — like Helm, Prometheus, and Grafana — can also be used to manage OpenEBS itself. The configuration of OpenEBS is completely controlled via YAMLs (Custom Resources) and volumes are provisioned via the Kubernetes Dynamic Provisioners, Storage Classes and Persistent Volume Claims.\n\nThe components of OpenEBS that act on the user requests (via CRs) and generate / launch the Kubernetes Objects — like Deployments, Services and Persistent Volumes are collectively called as the *Storage Management or Control Plane*. The interactions to the Storage Management Plane can achieved via `kubectl`.\n\nThe administrators can customize the OpenEBS parameters by modifying the YAMLs. Some of the configuration that are possible are:\n\n- Setting up Taints and Tolerations to decide the nodes where the Storage Pods need to be deployed.\n- Setting up the Node Selectors or Anti-Affinity parameters to select the correct nodes for launching the Volume (Data Engine) pods.\n\nFor further details check: [https://openebs.io](/docs?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n\nAnother cool feature that has become a favorite of advanced OpenEBS users is that, as Kubernetes introduces new features that have to be passed on to the Dynamically generated Volume (Data Engine) Pods, users can directly patch the pods and also update the Volume Template YAMLs at run time. To learn more about this feature, hit us up on slack: [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294)\n\nIn this regard, as we move towards 1.0, we would like to improve the documentation and governance around maintaining multiple versions of the API.\n\n**Security Controls:** Storage is a cluster add-on service. It needs to access the storage devices — either as hostpath or as block devices that are attached to the nodes. Since OpenEBS is completely Kubernetes native, access rights and privileges required by the individual components are completely transparent and can be controlled by the RBAC configuration by Kubernetes Cluster Administrators. One of the interesting feedback we received as Enterprises started to adopt OpenEBS was that — the IT Teams want to retain the control on the storage and shield the specifics of the node details or scheduling details from application developers.\n\nOpenEBS can now be configured easily to run in:\n\n- Security Enhanced Linux Platforms (selinux=on) like RHEL, CentOS or OpenShift. We talk in more detail about this [here](/blog/configuring-openebs-to-run-with-securityenhanced-linux?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n- OpenEBS can be used on clusters where the default setting is to not grant access to the hostpaths for Developer namespaces. To support this use case, we now support an option to run the Jiva Volumes Pods (that require access to hostpath) to be deployed in OpenEBS Namespace using a *StoragePolicy**—**DeployInOpenEBSNamespace*. The StoragePolicies in OpenEBS are configured via StorageClasses.\n- OpenEBS can be used on clusters enabled with Pod Security Policies. The PSP for OpenEBS has been contributed by a user and can be found [here](https://github.com/openebs/openebs-docs/issues/484).\n\n**Scale and Performance:** OpenEBS is architected to horizontally scalable with nodes — Persistent Volumes spreading out pretty evenly across the nodes or a subset of nodes designated for Storage. However the data of any given Persistent Volume is always fixed to a specific set of nodes, eliminating the need for expensive metadata lookups to find the data blocks as the number nodes get higher. What I like even better is, that the replica tells the controller that it has the data effectively inversing the responsibility from volume target/controller having to know where data is located to replicas telling I have the data.\n\nAnother aspect of OpenEBS Volumes when it comes to performance is that each Volume is completely isolated and doesn’t get impacted by work/load on other Volumes. For example, a node rebuild will not degrade ALL volumes in the cluster.\n\nAs OpenEBS is completely developed in user space and run as Kubernetes Pods, administrators get complete control on the resources like CPU/RAM that should be allocated to Storage. There won’t be cases of kernels hogging all the resources. Administrators also can tune for example the number of threads allocated and parallel IOs supported per Volume — tuning will have an impact depending on the type of workloads (Sequential / Random).\n\nOf course, then we have workloads that require low latency, and need to work to be deployed on nodes with limited storage available, for example a couple of NVMe devices. OpenEBS Local PVs provide the functionality to make use of the Local Storage and help with dynamic provisioning of Local PVs. OpenEBS Local PVs offer an excellent choice for cases like NuoDB where replication is inherently taken by NuoDB itself and the expectation is only to get a persistent storage with node affinity configured for the storage pods. A default storage class called — `openebs-hostpath` is available in the current release. Check it out and let us know what you think.\n\nWe are seeing OpenEBS users progress from running CI/CD workloads in staging to now running critical databases in production on OpenEBS Volumes. If you are interested in performance, we have a OpenEBS user currently exploring and sharing the benchmarking numbers on several different platforms. Join the discussion on our slack — [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n\n— -\n\nWhile I covered earlier some of the items introduced recently in [0.9](https://github.com/openebs/openebs/releases), here is a summary of some significant changes:\n\n- Introduction of Dynamically provisioned OpenEBS Local PVs for making use of the storage available on the nodes itself for running NewSQL kind workloads. Refer to this [blog](/blog/dynamic-provisioning-of-kubernetes-local-pvs-using-openebs?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294) to get started with OpenEBS Local PVs.\n- Enhanced the cStor Replica distribution logic for MongoDB or Cassandra Statefulsets to provide storage high availability and reducing the performance overhead\n- Backup and Restore processes for cStor Volumes using [OpenEBS Velero Plugin](https://github.com/openebs/velero-plugin) that can perform incremental snapshot backup and restore.\n- Enhance the Deployment and Placement of the Jiva Volume Pods to facilitate working in environments with strict Pod Security Policies and for Kubernetes clusters that tend to see a lot of pod evictions or node drains.\n- Introduced Web Admission Hook that will help with validation and avoiding misconfigurations.\n- Developed an upgrade framework using the CAS Templates that will help developing operator based upgrades from earlier releases to the current release. We are excited to use this framework for upgrading from 0.8.2 to 0.9\n- Enhanced the Prometheus exporters to support generating cStor Volume Replica metrics.\n- As always many other user reported issues made into this release. To learn more checkout the [release notes](https://github.com/openebs/openebs/releases).\n\nEach OpenEBS release goes through Litmus — GitLab Pipelines that verify new functionality, backward compatibility and also acceptance from users through pre-release build testing. Please reach out to us if you would like to be included in the pre-release notifications.\n\nI am very excited about the following active contributions that are filling in the gaps for 1.0\n\n- BDD tests that developers can execute as part of the feature development. This is an extension to the Sanity tests are executed in Travis CI and Litmus Tests in GitLab CI.\n- Updating the Design, Contributor and Governance related details with the help of CNCF guidelines.\n- Freezing on the cStor Specs based on the feedback received from users on how to make it more user friendly and easy to manage. The NDM Disk Specification has also changed to BlockDevice spec as per the feedback received in making it generic to Storage Devices.\n- Additional enhancements automatic the cStor Day 2 operations with regards to scaling up the capacity of cStor Pool by increasing the size of the underlying disks, scaling up and down the number of cStor Pools in a given cluster.\n- Support for integrating the OpenEBS Local PV into the BlockDevices discovered and managed by NDM.\n- Support for ARM builds.\n\nFor contributing to the above feature or learning more about them, you can reach out to us on the #contributors channel or check out the [milestones](https://github.com/openebs/openebs/milestones).\n\nI am very excited to be at KubeCon Europe and to meet in person some of the hundreds of contributors and many thousands of users of OpenEBS who have all made it possible for OpenEBS to be recently accepted as a CNCF Sandbox Project. Thank you Again!\n\nMayaData will be speaking about OpenEBS at the KubeCon events in Barcelona in the Cloud Native Storage Day and Open Data Autonomy Mini Summit on May 20th, and also at booth SE41 until Thursday in KubeCon Expo Hall.\n","notHasFeatureImage":false,"slug":"openebs-project-update-and-whats-coming-in-v1.0"},{"id":46,"title":"Configuring OpenEBS to run with Security-Enhanced Linux","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"10-05-2019","tags":["Centos","Kubernetes","OpenEBS","Selinux","Uncategorized","Openshift","Tutorials"],"excerpt":"In this short How-To blog, I will walk you through the steps for running OpenEBS with Security Enhanced Linux (SELinux=on).","content":"\nIn this short How-To blog, I will walk you through the steps for running OpenEBS with Security Enhanced Linux (SELinux=on). This applies to both OpenShift based Kubernetes Cluster or Clusters using CentOS.\n\nOpenEBS can be treated as a Cluster Add-on service, that can convert the existing Kubernetes Cluster into a full fledged Storage Controllers — that can serve Persistent Volumes to Stateful Workloads while making use of the storage attached to the nodes. Since OpenEBS runs in containers and has to access the underlying storage devices, it needs to run in Privileged mode.\n\nThe component of OpenEBS ( Node Device Manager or Node Disk Manager or simply referred as NDM), is responsible for discovery of the devices and filtering out devices that should not be used by OpenEBS; for example the disk that has OS filesystem. Earlier, to detect the OS disk, the NDM pod by default mounted the `/proc/1/mounts` file, which is restricted on nodes that have SELinux=on. This is now fixed by mounting the `/proc` directory of the host inside the container and then loading the `mounts` file.\n\nSo at a high level, to allow OpenEBS to run in privileged mode in SELinux=on nodes, the cluster should be configured to grant privileged access to OpenEBS service account.\n\nHere are the steps I have followed:\n\n**Step 1: Setup appropriate security context for OpenEBS**\n\n**On OpenShift Clusters:** Select the right SCC for OpenEBS\n\nOpenEBS like other cluster add-on services requires its pods to be executed with privileged access as it needs to directly access the block devices. However, this will not mean that SELinux should be turned off.\n\nWhile running on OpenShift, we need to be aware of the security constraints available and to select the right security constraint for OpenEBS. There are 7 SCCs available.\n\n![Security Context Constraints (SCC) in OpenShift](/images/blog/security-context-constraints-(SCC)-OpenShift.png)\n\nTo have OpenEBS pods running in privileged mode, add the OpenEBS service account (openebs-maya-operator) to use the privileged SCC; OpenShift will take care of setting the correct permissions (or SELinux labels) to make the container run in privileged mode. Sample commands to do this:\n\n oc adm policy add-scc-to-user privileged system:serviceaccount:openebs:openebs-maya-operator\n\n**On CentOS/RHEL:** Enable Pod Security Policies for OpenEBS\n\nCreate a file `openebs-privileged-psp.yaml` with the below spec.\n\n apiVersion: extensions/v1beta1\n kind: PodSecurityPolicy\n metadata:\n name: openebs-privileged\n annotations:\n seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'\n spec:\n privileged: true\n allowPrivilegeEscalation: true\n allowedCapabilities:\n - '*'\n volumes:\n - '*'\n hostNetwork: true\n hostPorts:\n - min: 0\n max: 65535\n hostIPC: true\n hostPID: true\n runAsUser:\n rule: 'RunAsAny'\n seLinux:\n rule: 'RunAsAny'\n supplementalGroups:\n rule: 'RunAsAny'\n fsGroup:\n rule: 'RunAsAny'\n\nThen apply the YAML file\n\n kubectl apply -f openebs-privileged-psp.yaml\n\n**Step 2: Install OpenEBS**\n\nDownload the latest version of `openebs-operator.yaml` file.\n\n wget https://openebs.github.io/charts/openebs-operator-0.8.2.yaml\n\n**On CentOS/RHEL:** An extra step of adding PSP to the Cluster Role is required.\n\nEdit the ClusterRole in the YAML to add `openebs-privileged` PSP\n\n # Define Role that allows operations on K8s pods/deployments\n kind: ClusterRole\n apiVersion: rbac.authorization.k8s.io/v1beta1\n metadata:\n name: openebs-maya-operator\n rules:\n - apiGroups: [\"*\"]\n resources: [\"nodes\", \"nodes/proxy\"]\n verbs: [\"*\"]\n - apiGroups: [\"*\"]\n resources: [\"namespaces\", \"services\", \"pods\", \"deployments\", \"events\", \"endpoints\", \"configmaps\", \"jobs\"]\n verbs: [\"*\"]\n - apiGroups: [\"*\"]\n resources: [\"storageclasses\", \"persistentvolumeclaims\", \"persistentvolumes\"]\n verbs: [\"*\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshots\", \"volumesnapshotdatas\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"apiextensions.k8s.io\"]\n resources: [\"customresourcedefinitions\"]\n verbs: [ \"get\", \"list\", \"create\", \"update\", \"delete\"]\n - apiGroups: [\"*\"]\n resources: [ \"disks\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"storagepoolclaims\", \"storagepools\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"castemplates\", \"runtasks\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"cstorpools\", \"cstorvolumereplicas\", \"cstorvolumes\"]\n verbs: [\"*\" ]\n - apiGroups: ['extensions']\n resources: ['podsecuritypolicies']\n verbs: ['use']\n resourceNames:\n - openebs-privileged\n - nonResourceURLs: [\"/metrics\"]\n verbs: [\"get\"]\n\nInstall OpenEBS\n\n kubectl apply -f openebs-operator-0.8.2.yaml\n\n**Note: If you are using helm to install openebs, you will need to apply the above change after it has been installed. In a future release of the helm chart, I will work on making this configurable parameter.**\n\n**Step 3: (Optional) Create a new cStor Pool.**\n\nYou can skip this step if using the default cStor Sparse pool.\n\n**Step 3a**: Verify all pods are working and cStor Pools are running\n\n![List of all pods in openebs namespace after installation](/images/blog/pod-lists.png)\n\n**Step 3b**: Verify that disks available on the nodes are discovered.\n\n kubectl get disks\n\n\n![Disks detected by NDM, along with sparse disks](/images/blog/ndm-detected-disks.png)\n\n**Step 3c**: Create a storage pool claim using the instructions at [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools)\n\nCreate a `cstor-pool-config.yaml` as mentioned in the docs.\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-pool1\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 1Gi\n - name: PoolResourceLimits\n value: |-\n memory: 2Gi\n spec:\n name: cstor-pool1\n type: disk\n maxPools: 3\n poolSpec:\n poolType: striped\n disks:\n diskList:\n - disk-301722dba60542a05ec2817773ff11ed\n - disk-a2d99f96090f0675b7ea60925b58414d\n - disk-cd01d33213c3a42c6b1a9f0798277368\n\nApply this file `kubectl apply -f cstor-pool-config.yaml`\n\n![3 cStor pool pods will be running](/images/blog/cstor-pool.png)\n\n**Step 3d**: Create a new storage class using SPC as `cstor-pool1` or edit the default storage class to use the newly created SPC. I have edited the already available default storage class.\n\n**Step 4: Running Percona Application**\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-cstor-sparse-deployment.yaml\n\nEdit the file to remove security context from the percona deployment.\n\nApply the yaml file\n\n kubectl apply -f percona-openebs-cstor-sparse-deployment.yaml\n\n\n![The percona cstor pod up and running in default namespace](/images/blog/percona-pod.png)\n\nThe `percona-cstor` pod will be up and running.\n\nThanks for reading my blog! We continue to look for feedback and suggestions on how we can improve OpenEBS integration with all possible platforms and workloads. If you have any feedback on my blog or see any area of improvement with OpenEBS in general, please provide feedback below or find us on Twitter or on the OpenEBS slack community [http://slack.openebs.io](http://slack.openebs.io/).\n","notHasFeatureImage":false,"slug":"configuring-openebs-to-run-with-security-enhanced-linux"},{"id":47,"title":"How to Easily Build a CI Dashboard","author":"Chandan Kumar","author_info":"Software Engineer at MayaData Inc","date":"26-04-2019","tags":["CI Dashboard","Git","Kubernetes","Pipeline","GitLab"],"excerpt":"In this tutorial, we’ll go through all the necessary steps for setting up the CI dashboard","content":"\nA code that is never executed for users is essentially a digital waste product. To prevent this building of waste and to showcase the results of code on the Kubernetes environment, we can use the CI Dashboard, along with chaos testing.\n\nThe CI dashboard allows users to view the commit and release-based build and run a chaos test using litmus on a different platform with different versions of Kubernetes.\n\nIn this tutorial, we’ll go through all the necessary steps for setting up the CI dashboard:\n\n1. *Create a project*\n2. *Push the code to GitHub*\n3. *Setup the CI of the project using GitLab*\n4. *Select the chaos test from [*Litmus*](https://github.com/openebs/litmus)*\n5. *Add the script in GitLab YAML to create pipelines for executing the chaos tests*\n6. *Build a CI dashboard(ex: *[*openebs.ci*](https://openebs.ci/)* ) to display the gitlab pipeline history and status*\n7. *Conclusion*\n\n### Step 1: Create a project\n\nFirst, create a [project](https://github.com/openebs/maya) and write some automated testing for it. You should also add the Dockerfile in the project to set up the CI.\n\n### Step 2: Put the codes on GitHub\n\nCreate a repository on GitHub and add a `.gitignore` file to ignore the auto-generated folder or file. Follow the script below to put changes into GitHub.\n\n $ git init\n $ git add .\n $ git commit -s -m \"Initial commit\"\n $ git remote add origin .git\n $ git push origin master\n\n### Step 3: Setup the CI using gitlab\n\nAdd a *.gitlab.yaml* file to project and write the build and test steps. \n(Ex: [https://github.com/openebs/maya/blob/master/.gitlab-ci.yml](https://github.com/openebs/maya/blob/master/.gitlab-ci.yml)). Import the project in gitlab from GitHub. Setup the gitlab pipeline environment variable to push the docker image, or any other, if required. Add the pipeline trigger command in *.gitlab.yaml* file.\n\n### Step 4: Selection of chaos test\n\nSelect the chaos test (litmus book) from the [litmus](https://github.com/openebs/litmus) repository or write your own litmus book if needed. This will be used to test the product performance on different Kubernetes versions and with different cloud vendors.\n\n### Step 5: Add script in gitlab.yaml\n\nCreate a [repository](https://github.com/openebs/e2e-packet) for the execution of the platform-based pipeline. Add `.gitlab.yaml` file and related script to create a cluster, or use the executing cluster and run the different chaos tests in various stages of the [pipeline](https://gitlab.openebs.ci/openebs/e2e-packet/pipelines).\n\nReference the `.gitlab.yaml` file\n[https://raw.githubusercontent.com/openebs/e2e-packet/master/.gitlab-ci.yml](https://raw.githubusercontent.com/openebs/e2e-packet/master/.gitlab-ci.yml)\n\n cleanup-packet:\n when: always\n image: chandankumar4/packet:v4\n dependencies:\n - packet-cluster\n stage: CLUSTER-CLEANUP\n script: \n - chmod 755 ./script/packet-cleanup\n - ./script/packet-cleanup\n \n\n### Step 6: Build a CI dashboard\n\n![CI Dashboard](https://lh4.googleusercontent.com/hoDf2G6VnpIhhkmkQXlF07ocFRm7bJjP5f1ZkA8TZCT6PXMOPkdCO966EecYpk7koCbHPKdMemOA3_kYz8M5qrvLevRDJPw2c0MfYn-yp-iLn4j-qV8wpwT_av2iBYBuMH-4EUeB)\n\nCreate a project called [Ci dashboard backend](https://github.com/openebs/ci-e2e-dashboard-go-backend) that will fetch the pipeline details from gitlab by accessing their API and exposing the same on different API after some enhancement. Create another project called [Ci dashboard](http://github.com/openebs/ci-e2e-dashboard) that will display the gitlab pipeline details by accessing the data from the back end API.\n\n### Step 7: Conclusion\n\nCI dashboard will display the build history of the imported project and analyze the performance on different platforms and versions of Kubernetes.\n\n### References\n\n[https://openebs.ci/](https://openebs.ci/)\n\n[openebs/ci-e2e-dashboard](https://github.com/openebs/ci-e2e-dashboard)\n[Contribute to openebs/ci-e2e-dashboard development by creating an account on GitHub.github.com](https://github.com/openebs/ci-e2e-dashboard)\n\n[openebs/ci-e2e-dashboard-go-backend](https://github.com/openebs/ci-e2e-dashboard-go-backend)\n[OpenEBS CI Dashboard backend using Go and PostgreSQL. — openebs/ci-e2e-dashboard-go-backendgithub.com](https://github.com/openebs/ci-e2e-dashboard-go-backend)[openebs/maya](https://github.com/openebs/maya)\n[OpenEBS Maya extends Kubernetes capabilities to orchestrate CAS containers. — openebs/mayagithub.com](https://github.com/openebs/maya)\n","notHasFeatureImage":false,"slug":"how-to-easily-build-a-ci-dashboard"},{"id":48,"title":"Deploying OpenEBS on SUSE CaaS platform","author":"Ashok Babu","author_info":"Senior DevOps consultant at Wipro works on App Anywhere & cloud-native technologies.","date":"24-04-2019","tags":["Kubernetes","Cloud Native Storage","MayaData","SUSE","Container attached storage","OpenEBS"],"excerpt":"I am recently introduced to OpenEBS, an easy to use persistent storage option for Kubernetes and found it nicely working on the SuSE CaaS platform where I could certify the cloud native databases. ","content":"\nI am recently introduced to OpenEBS, an easy to use persistent storage option for Kubernetes and found it nicely working on the SuSE CaaS platform where I could certify the cloud native databases. In this blog, I cover a few quirks to get it running on the SuSE CaaS platform.\n\nSuSE CaaS Platform is an enterprise-class container management solution that leverage Kubernetes as the orchestration layer and SuSE MicroOS as the host operating system for master and worker nodes.\n\nSuSE CaaS provides enhanced security policies such as predefined pod security policies.\n\nIn SuSE MicroOS, a read-only Btrfs file system is used for the root file system with OverlayFS. Sub-volumes for data sharing is read-write.\n\nOutlined below are some of the steps that need to be taken care while installing OpenEBS version 0.8.1 on SuSE CaaS Platform 3\n\n## Issue\n\nDefault OpenEBS installation would fail on SuSE CaaS platform due to the following restrictions by the platform\n\n- NDM Daemonset fails to spin up as it requires privilege permission\n- Sparse pools will not be created as it uses by default /var/openebs directory which is read-only directory under root filesystem in SuSE CaaS.\n- Runtasks in operator uses /var/openebs directory for temporary file creation due to this cstor-target pods or cstor-pool pod gets stuck in “ContainerCreating” status\n\n## Resolution\n\n### Step1\n\nTo install OpenEBS on SuSE platform run the following custom YAML file instead of default operator YAML\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-susecaas.yaml\n\nThis is a customized operator YAML file for SuSE CaaS platform that uses the role `suse:caasp:psp:privileged` for creating privileged DaemonSet\n\n[https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#ex.admin.security.pod_policies.daemonset](https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#ex.admin.security.pod_policies.daemonset)\n\n### Step2:\n\nPerform the following changes in the runtask after completing step1\n\nNote: — This step is not required if you are using the OpenEBS version 0.9 which is the upcoming release.\n\n- **Change the path in runtask for cstor-pool-create-putcstorpooldeployment**\n\n kubectl edit runtask cstor-pool-create-putcstorpooldeployment-default-0.8.1 -n openebs\n\n Change from\n\n path: /var/openebs/shared-\n\n Change to\n\n path /var/lib/overlay/openebs/shared-\n\n- **Change the path in runtask for cstor-volume-create-puttargetdeployment**\n\n kubectl edit runtask cstor-volume-create-puttargetdeployment-default-0.8.1 -n openebs\n\n Change from\n\n path: /var/openebs/shared--target\n\n Change to\n\n path: path /var/lib/overlay/openebs/shared--target\n\n### Step 3:\n\n**Optional:** If you need to use sparse pool\n\n kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/5860c0a4619a9feddf5d75d11f50f2ea8fdcec82/k8s/demo/fio/demo-cstor-sparse-pool-limits.yaml\n\n### Step 4:\n\nConfiguration of storage pool, storage class and PVC are like any other platform and the steps are outlined in [https://openebs.io/docs](/docs?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294)\n\nPool Configuration — [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#manual-mode)\n\nStorage class — [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-class](/docs/deprecated/spc-based-cstor#creating-cStor-storage-class?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#creating-a-new-class)\n\nVolume — [https://openebs.io/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume](/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#provision-from-a-disk-pool)\n\n## Conclusion:\n\nAbove approach can be followed to install OpenEBS 0.8 on SuSE CaaS platform 3.0 which needs additional configuration.\n\nWith OpenEBS 0.9 it would be simplified such that you would need to only apply the operator YAML to perform the installation.\n\n**SUSE CaaS Platform:** (Container as a Service Platform) is an integrated software platform which automates the process of building, managing and upgrading of Kubernetes clusters. It combines the benefits of an enterprise-ready operating system with the agility of an orchestration platform for containerized applications. More details — [https://www.suse.com/products/caas-platform/](https://www.suse.com/products/caas-platform/)\n\n**OpenEBS:** OpenEBS is the leading open-source project for container-attached and container-native storage on Kubernetes. OpenEBS adopts Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller. OpenEBS implements granular storage policies and isolation that enable users to optimize storage for each specific workload. OpenEBS runs in userspace and does not have any Linux kernel module dependencies. More details — [https://openebs.io/](https://openebs.io/?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294)\n\nThey have a very responsive community. Visit [https://slack.openebs.io](https://slack.openebs.io)\n","notHasFeatureImage":false,"slug":"deploying-openebs-on-suse-caas-platform"},{"id":49,"title":"High Availability For Jira Using OpenEBS","author":"Patrick Hoolboom","author_info":"Works @MayaData","date":"21-04-2019","tags":["OpenEBS","Kubernetes","K8s","Jira","Docker","Mayaonline"],"excerpt":"Up time, high availability, disaster recovery. Anyone that has been working in development or IT operations roles knows these terms. Maintaining these things is always touted as the most critical of critical tasks.","content":"\nUp time, high availability, disaster recovery. Anyone that has been working in development or IT operations roles knows these terms. Maintaining these things is always touted as the most critical of critical tasks. Even with that in mind, we often slack in these areas. We don’t test our backups or run our applications with high availability in mind. One of the main factors that lead to these less than ideal deployments is that implementing a good high availability story, or testing our backups, adds time and/or complexity to our already busy day. All that being said, let’s take a look at a dead simple solution for one of those applications that many of us run: Jira.\n\nThe standard answer to making Jira highly available and provide a good disaster recovery story is to use Jira Data Center and follow this guide:\n\n[Jira Data Center Guide](https://confluence.atlassian.com/enterprise/jira-data-center-472219731.html)\n\nThat guide shows you how to cluster multiple instances of the application behind a load balancer, then use a shared file system and a shared database underneath. This blog is not about the application layer or even the database (though we do have some fantastic guides for deploying databases using OpenEBS in our documentation). This is going to focus on a simple way to create a replicated storage layer specifically for the file system. Jira stores information on disks such as issue attachments, import/export files, and logos. These are all important in keeping Jira working correctly.\n\nDeploying Jira on Kubernetes using OpenEBS is as simple as installing OpenEBS on Kubernetes, define a storage pool, define a storage class, define your persistent volume claim, and deploy the Jira container. That’s it… and if you are already using OpenEBS it is even simpler. Now, as for the specifics of how to do those things, see this guide:\n\n[Jira - OpenEBS docs](/docs/stateful-applications/jira?__hstc=216392137.fb75a0ac1e54cb037dfbafd0edf1ad3f.1579868085240.1579868085240.1579868085240.1&__hssc=216392137.1.1579868085240&__hsfp=3765904294)\n\nBy leveraging the power of replicas within OpenEBS we have quickly added fault tolerance to the storage attached to our Jira container. It was as simple as defining a replica count in the storage class. Since the replicas are spread across the cluster we no longer have to worry about the storage being a single point of failure. If one disk goes down, the controller will automatically route to one of the replicas with no intervention necessary.\n\nAs you can see OpenEBS has greatly simplified the process of making Jira more resilient in a containerized world. The ease of use of container attached storage makes tasks like these much simpler. It allows us to spend more time working on improving our applications or infrastructure, and less time worrying about its resiliency.\n","notHasFeatureImage":false,"slug":"high-availability-for-jira-using-openebs"},{"id":50,"title":"A Primer on OpenEBS Continuous Integration","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"02-04-2019","tags":["Chaos Engineering","Continuous Integration","E2e Kubernetes Litmus"],"excerpt":"The OpenEBS project consists of several components (control plane & data plane) that directly support the dynamic provisioning & management of a persistent storage volume on the Kubernetes cluster.","content":"\n**This article is part of a #Concepts series on Kubernetes and Litmus. Here, we present an overview of the OpenEBS CI infrastructure with its Litmus powered e2e pipelines.**\n\nThe OpenEBS project consists of several components (control plane & data plane) that directly support the dynamic provisioning & management of a persistent storage volume on the Kubernetes cluster. As with any microservice-oriented system following the DevOps paradigm, there is a need to continuously build and test each component, both in isolation (via unit tests) as well as in relation with the other pieces (integration tests) with emphasis on standard end-user scenarios (e2e). Factor in the need for basic interoperability verification (in terms of supported OS/Platform/Cluster versions) and you have the requirements for the CI framework spelt out.\n\nThe OpenEBS CI infrastructure is based on the Cloud-Native GitLab CI framework which is setup to monitor commits to the core components such as [Maya](https://github.com/openebs/maya), [Jiva ](https://github.com/openebs/jiva)& [cStor](https://github.com/openebs/zfs) and also [e2e ](https://github.com/openebs/e2e-packet)(reduces turnaround time to verify test case sanity). It uses Litmus to drive the e2e pipelines, thereby providing a reference implementation for a Litmus-Experiment based e2e suite.\n\n## GitLab Infrastructure\n\nSome of the reasons for adopting GitLab as the CI framework of choice (amongst standard benefits such as tight integration with our existing SCM, 2-factor auth, webhook support, well-defined UI with pipeline graphs, etc was the need to move away from a plugin-based model (Jenkins thrives on plugins, which may not always be advantageous) to a self-contained platform that supports simple pipeline definitions (`.gitlab-ci.yaml` is far easier to maintain than the groovy-based `jenkinsfile`!). GitLab also offers a more mature kubernetes-native solution that gives users the ability to dogfood OpenEBS storage as the back-end store (PostgreSQL) for the GitLab server.\n\n![GitLab server](/images/blog/openebs-control-and-data-plane.png)\n\nThe GitLab server (with its microservices such as Unicorn, Shell, Workhorse, Registry, Sidekiq, Gitaly, PostgreSQL, Redis, Minio) is hosted on a multi-node bare-metal OpenShift cluster and is configured with pull-based repository mirroring of the OpenEBS component GitHub repos and a webhook based setup that triggers the pipelines upon commits.\n\nWhile Maya and Jiva repos are mapped to shell-based executors due to certain build and integration-test requirements, cStor (zfs) and e2e repos are mapped to docker-machine based executors. The docker-machine executors are inherently auto-scaling, a necessary feature for e2e builds as multiple parallel jobs are spawned during the course of e2e pipelines.\n\n## OpenEBS CI Workflow\n\nEach commit to the component source triggers the “gitlab build” procedure, which can be split into two logical phases: “component build” and “e2e.”\nThe component build executes the respective makefile which typically involves running unit tests, building the GO binaries, creating docker images, running integration tests and pushing docker images tagged with the commit SHA to the respective container repositories. It also performs certain pre-e2e routines before finally triggering the e2e pipelines.\n\nThe e2e phase involves running several parallel pipelines (based on Kubernetes cluster platforms or versions), with each pipeline containing multiple stages such as test bed setup, application deployment, Litmus experiments (functional and chaos), and finally clean-up. Needless to say, the component versions used are the ones built in the previous phase.\n\n![OpenEBS CI Workflow](/images/blog/openebs-ci-flow.jpg)\n\n**Note**: *Currently, the GitLab CI works in a “retrospective mode,” as it is invoked on commits to the upstream branches. There is work in progress to extend the support for pull requests (presently, a travis-based build verifies commit sanity to aid PR acceptance).*\n\n## Baseline Commit\n\nBefore triggering the e2e pipelines as part of the final step, the build pipeline performs a pre-e2e routine to generate metadata about the impending e2e run. As is evident from the previous discussion, the e2e pipelines are triggered against commits to any of the OpenEBS component repos (maya, jiva, zfs/cstor, e2e). Images pushed as part of the build pipeline are deployed during the e2e. Therefore, it is necessary to baseline or identify an e2e run against the primary trigger (commit) while maintaining the details of image versions of the other relative components.\n\nThis is achieved by writing the details of the baseline commit (timestamp, component repo, branch and commit ID) into the file head of a “baseline artifact” maintained in a separate repository. Once the e2e pipeline is initiated, the baseline artifact is [parsed ](https://github.com/openebs/e2e-infrastructure/blob/master/env/env_exporter.py) for the “most current” image tags of each component, which will invariably include the current baseline commit and the latest ones for other components, in the test-bed preparation stage. This information is then used to [precondition ](https://github.com/openebs/e2e-infrastructure/blob/master/env-update/env.py) the OpenEBS Operator manifest before its deployment on the test clusters.\n\n## OpenEBS e2e Pipelines: Leveraging Litmus\n\n*One of the arguments against the inclusion of e2e as part of CI pipelines is that they could be flaky and time-consuming (under ideal circumstances it involves testing every moving part of the microservice and needs more maintenance). However, the e2e can confirm achievement of the goals that the solution was conceived and built to accomplish. The extent of coverage and the schedules can be optimized for development agility, but at OpenEBS, we feel it is a must-have in our CI pipelines.*\n\nOpenEBS CI makes use of Litmus to drive its e2e pipelines, right from test bed creation (cluster creation playbooks) all the way through the e2e tests (litmus experiments). The e2e pipeline involves several stages, with one or more gitlab jobs scheduled to run in a given stage. Each gitlab job is associated with a “runner script” that runs an “e2e test.” This in turn invokes/executes a litmus experiment (or litmus ansible playbook in the case of cluster creation/destroy jobs).\n\nThe various stages in the e2e pipeline are discussed below:\n\n![Various stages in the e2e pipeline](/images/blog/e2e-pipelines.png)\n\n**Cluster Creation**: This stage calls up the Kubernetes cluster by executing the platform-specific playbooks. Cluster parameters are controlled via runtime arguments. The artifacts generated upon this job’s execution such as cluster config, which includes kubeconfig and cluster resource names, are passed over to subsequent stages as dependencies. The Litmus pre-requisites are also installed once the cluster is created. Currently, Litmus supports creation of clusters on these platforms:\n\n- Baremetal Cloud: Packet (ARM based physical servers)\n- Managed Kubernetes: GKE, EKS, AKS\n- Cloud Hosted/Self-Installed: AWS, GCP (via KOPS)\n- On-Premise: OpenShift (vSphere VMs)\n\n**Cluster Provision**: Provision equips the cluster with additional disk resources native to the specific platform (GPD, EBS, Packet Block Storage, Azure Block Device) used by the storage-engines as physical storage resources.\n\n**Provider Setup:** Here, the system deploys the customized/preconditioned OpenEBS Operator manifest (based on the baseline commit) on the cluster, thereby setting up the control plane and preparing default storage pool resources. The logging infrastructure (fluentd) is also setup on the created cluster.\n\n**Stateful Application Deployment**: The OpenEBS e2e verifies interoperability with several standard stateful applications such as Percona-MySQL, MongoDB, Cassandra, PostgreSQL, Prometheus, Jenkins, Redis etc. These applications are deployed with OpenEBS storage classes (tuned for each app’s storage requirement). Each application is accompanied by respective load-generator jobs that simulate client operations and real-world workloads.\n\n**App Functionality Tests:** Each deployed application is subjected to specific behavioural tests such as replica scale, upgrade, storage resize, app replica re-deployment, storage affinity etc. Most of these tests are common day-2 operations.\n\n**Storage/Persistent Volume Chaos Tests:** The PV components such as controller/replica pods are subjected to chaos (pod crash/kill, lossy networks, disconnects) using tools such as ChaosKube, Pumba, and Kubernetes APIs (via kubectl) to verify data availability and application liveness during adverse conditions.\n\n**Infrastructure Chaos Tests:** The cluster components such as storage pools, nodes, and disks are subjected to different failures using Kubernetes APIs (forces evicts, cordon, drain) as well as platform/provider specific APIs (gcloud, awscli, packet) to verify data persistence and application liveness.\n\n**Stateful Application Cleanup:** The deployed apps are deleted in this stage, thereby verifying de-provisioning and cleanup functionalities in the OpenEBS control plane.\n\n**Cluster Cleanup:** The cluster resources (nodes, disks, VPCs) are deleted. With this final step, the e2e pipeline ends.\n\n## GitLab e2e Job Runner Template\n\nEach GitLab job running the e2e test executes (bash) scripts containing steps to run and monitor a Litmus experiment. These scripts are invoked using desired arguments, specified as part of the job definition in the e2e repository’s `.gitlab-ci.yml`. The standard template maintained in these (bash) runner scripts and the performed tasks are described below.\n\n![GitLab runner template(E2E)](/images/blog/gitlab-job-runner.png)\n\n**Generate Unique Test Name:** Each gitlab job is associated with a litmus experiment that has a test/experiment name. The result of this litmus experiment is stored in a Litmus Custom Resource (CR) of the same name. The success of a test and therefore the gitlab job is derived from this CR. Occasionally, it is possible that the same litmus experiment is run against different applications or storage engines in a pipeline, thereby necessitating a unique element or ID in the CR name. In this step, a user-defined input (run_id) is accepted to generate a unique test/CR name.\n\n**Setup Dependencies:** Depending on the nature of the gitlab job (cluster create/delete playbooks OR litmus experiments), the executor machine is updated with the appropriate directory structure and target cluster info (such as cluster configuration file, cluster names, disk details, VPC information etc.) to ensure successful test execution.\n\n**Precondition Litmusbook:** Each litmusbook (the Kubernetes job specification YAML) consists of a default set of test inputs such as placeholders for application, storage, chaos info, etc. These are overridden/replaced by desired values in this step. In addition, the default name of the test is replaced with the unique name generated by the runner at the start of execution.\n\n**Run Litmus Experiment:** The litmusbook is deployed and monitored for completion, with a polling interval of 10s. The status of both the litmus Kubernetes job and the ansible-runner container is checked as necessary and sufficient conditions, respectively, to determine completion of the litmus experiment.\n\n**Get Litmus Experiment Result:** The result CR with the unique name generated is queried to determine the Litmus experiment result. The runner script completes execution with a zero/non-zero exit code depending on a pass/failure result, thereby setting the gitlab job status.\n\n**Logging Framework for e2e**\n\nOpenEBS CI uses the popular EFK (Elasticsearch-Fluentd-Kibana) stack as the logging framework for the e2e pipelines. Each target cluster brought up as part of the e2e pipeline is configured with the fluentd-forwarder daemonset and fluentd-aggregator deployment, with the latter streaming the logs to the remote ElasticSearch instance running on the master (GitLab-CI) cluster. These are then rendered by the Kibana visualization platform that is also running on the master cluster. The forwarders are configured with the appropriate filters based on pipeline and commit IDs to aid in a quick data analysis.\n\n![EFK-Based Logging framework](/images/blog/efk-based-logging-framework.png)\n\n**Conclusion**\n\nHopefully, this article has provided you with a better understanding of the CI/E2E practices in the OpenEBS project. CI is an important factor in contributor happiness as well as user confidence, and we are focused on continually making it more robust. Feel free to share your questions, comments, and feedback with us — we are always listening!\n","notHasFeatureImage":true,"slug":"a-primer-on-openebs-continuous-integration"},{"id":51,"title":"Dynamic provisioning of Kubernetes Local PVs using OpenEBS","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-03-2019","tags":["Kubernetes","Open Source","OpenEBS","Stateful Applications","Storage"],"excerpt":"In this blog I discuss why we are adding the support of Local Persistent Volumes to the open source OpenEBS -  I view this as a significant step forward in our mission of delivery true data agility to the broader cloud native community and I welcome your feedback.","content":"\n*Updated May 16th 2019: The alpha version of the OpenEBS Local PV provisioner has been included in OpenEBS Release 0.9. This blog is updated with the setup instructions and examples from v0.9.*\n\nIn this blog I discuss why we are adding the support of Local Persistent Volumes to the open source OpenEBS — I view this as a significant step forward in our mission of delivery true data agility to the broader cloud native community and I welcome your feedback.\n\nThere are many stateful workloads like Mongo, Redis, Cassandra, Postgres, Nuodb etc.,that are capable of performing their own replication for high availability and basic data protection, eliminating the need for the underlying storage to copy or replicate the data for these purposes. Instead, what users of these workloads seem to really want from underlying storage is maximum throughput that can be delivered along with of course the guarantee that data is consistent on the disk — and increasingly we are seeing that the operators of these workloads also want to use Kubernetes as their fundamental API irrespective of what IaaS or hardware or Kubernetes service underlies their environment.\n\nKubernetes provides an option to use Host Paths (and very recently the use of Local PVs is also gaining popularity) which present a very simple and elegant option to get started. The Local PVs differ from the Host Paths in two aspects:\n\n- Application developers have to specify the Host Path in their Pod Spec.\n- Host Paths typically share a single underlying disk, where as Local PVs occupy the entire disk.\n\nThere are many scenarios where Host Paths are the only option since the Kubernetes Nodes can have a limitation on the number of Disks (or devices) that can be attached — either due to node capabilities or just for cost purposes.\n\nHowever, for Kubernetes Cluster administrators — it can soon become a nightmare if they give free reign to their application developers to use Host Path volumes — as can be seen from a recent conversation in the Kubernetes Slack.\n\n![slack conversation](/images/blog/slack-conversation.png) \n(***Issues - Slack conversation***)\n\nTypically Kubernetes Administrators lockdown (or disable) the `hostPath` feature using Pod Security Policies, especially in enterprises where security policies trump everything else.\n\nInterestingly, most of these challenges with the Host Path volumes can be overcome, by using a Dynamic Local PV Provisioner (managed by StorageClasses and PersistentVolumeClaims) that can be configured to create a Local PV with either a subpath or the entire disk.\n\nThe Kubernetes Cluster Administrators can reassert control by specifying via StorageClasses where and how the Host Path can be created and who can actually use them in the Cluster (by setting RBAC) policies.\n\nBesides overcoming the security issues with hostPath and gaining more control on how they are provisioned, Local PVs based provisioning of host paths has the following additional benefits:\n\n- Ability to pin the pods using the hostPath Local PV, by specifying the node selector. If native hostpath is used, there is no guarantee that the pod will get rescheduled to the same node in case of node reboots or pod eviction scenarios.\n- Ability to provide hooks that can either retain the PV and its data or perform clean up of the directory once the associated PVC is deleted.\n- Ability to provide exclusive ( ReadWriteOnce) or shared (ReadWriteMany) to the data on the host path.\n\nUsing Dynamic Provisioner for Local PVs also makes it easy to provide additional features of forthcoming Kubernetes Versions, such as:\n\n- Metrics Support\n- Enabling Capacity and PVC Resource Quotas\n- Full benefits of Volume Topology\n\n**Ready to give OpenEBS Local PVs a spin.**\n\n*OpenEBS Local PVs (alpha) is available in OpenEBS 0.9 release. Use the following commands to try it out.*\n\n*Step 1:* You need to have Kubernetes 1.12 or higher. Install the OpenEBS Local PV Provisioner.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.9.0.yaml\n\nNote: The above will setup the complete OpenEBS Control Plane along with the OpenEBS Dynamic PV Provisioner. Ensure that the ` openebs-localpv-provisioner` pod is running. `kubectl get pods -n openebs`\n\nThis version supports creating hostpath based Local PVs. A default StorageClass ` openebs-hostpath` is also loaded, that will create the host paths under `/var/openebs/local/`. If you would like to change the path, modify the StorageClass accordingly and re-apply.\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-hostpath\n annotations:\n openebs.io/cas-type: local\n cas.openebs.io/config: |\n - name: StorageType\n value: \"hostpath\"\n - name: BasePath\n value: \"/var/openebs/local/\"\n provisioner: openebs.io/local\n volumeBindingMode: WaitForFirstConsumer\n reclaimPolicy: Delete\n\n*Step 2:* Launch a MongoDB StatefulSet that makes use of Local Persistent Volumes.\n\n kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.9.x/k8s/demo/mongodb/demo-mongo-localpvhostpath.yaml\n\nCheckout that Local PVs are dynamically created — `kubectl get pv`.\n\nAlso go ahead and check that deleting the Pod will put it right back on the node where the PV was created.\n\n**Show me Code**\n\nIt is being cooked [here](https://github.com/openebs/maya/tree/master/cmd/provisioner-localpv). Any help with code review or suggestions welcome.\n\n**Credits!**\n\nThe implementation has been inspired by the feedback from the OpenEBS community who have requested for this feature to be available and used for Jiva Data engine and by all the prior work done by the Kubernetes Community.\n\n- [https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/tree/master/examples/hostpath-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/tree/master/examples/hostpath-provisioner)\n- [https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner)\n- [https://github.com/rancher/local-path-provisioner](https://github.com/rancher/local-path-provisioner)\n- [https://docs.openshift.com/container-platform/3.7/install_config/configuring_local.html](https://docs.openshift.com/container-platform/3.7/install_config/configuring_local.html)\n\n**What Next.**\n\nOpenEBS v1.0 continues to make significant progress in enhancing capabilities to the existing Storage Engines like ability to expand cStor Pools by adding additional disks, ability to replace a failed disk without affecting the applications, ability to resize the cStor volumes and many more. OpenEBS Local PVs will provide an additional option for the DevOps teams to cost effectively run their Stateful workloads.\n\nOpenEBS Dynamic Provisioner for Local Persistent Volumes works in tandem with the capabilities that are provided already by the OpenEBS Node Device Manager (NDM).\n\n![openebs local pv](/images/blog/openebs-local-pv.png)\n(***OpenEBS Local PVs***)\n\nThe Disk/Device Management Layer is provided by the OpenEBS NDM and projects built using Restic, that help with:\n\n- Discovering the devices attached to the Nodes\n- Performing Backup / Restore of Data from subpaths on the Nodes\n- Monitoring the devices used to provide the Host Paths, and in case the devices are backed by Disks — monitoring the SMART metrics to provide predictive error alerts.\n\nI believe this also opens up a lot of use cases to utilize the Capacity from the existing storage servers that are at the Users disposal by adding them to the Kubernetes Nodes and not having to worry about upgrading those SANs to support or wait for CSI specs to be implemented.\n\nWould love to hear your feedback on this feature. Do you use hostPaths or Local PVs that are natively supported in Kubernetes? What are your biggest challenges using those features and how can OpenEBS help? Please leave a comment in this blog or hit me up on the [Slack](https://slack.openebs.io/?__hstc=216392137.09f8198be7856f2c54869404c1891309.1580118253605.1580118253605.1580118253605.1&__hssc=216392137.1.1580118253605&__hsfp=3765904294).\n\n[Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":false,"slug":"dynamic-provisioning-of-kubernetes-local-pvs-using-openebs"},{"id":52,"title":"Block volume support for OpenEBS volume","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"27-03-2019","tags":["Block","Iscsi","Kubernetes","OpenEBS","Volume"],"excerpt":"A block volume is a volume that appears as a block device inside the container and allows low-level access to the storage without intermediate layers, as with file-system volumes.","content":"\nBy extending the API for *PersistentVolumes* to specifically request a raw block device, k8s provides an explicit method for volume consumption. Previously, any request for storage was always fulfilled using a formatted *filesystem*, even when the underlying storage was block storage. Kubernetes v1.13 moves raw block volume support to beta and allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system.\n\nIn addition, the ability to use a raw block device without a *filesystem* will give Kubernetes better support for high-performance applications that are capable of consuming and manipulating block storage for their needs. Block volumes are critical to applications such as databases (MongoDB, Cassandra) that require consistent I/O performance and low latency.\n\n## Block Volume Provisioning\n\nIn order to create a block volume, provisioners need to support *volumeMode*, then create a persistent volume with the desired volumeMode. If the admin selects an external provisioner that is capable of provisioning both filesystem and block volumes, he/she will have to carefully prepare the Kubernetes environment for their users as it is necessary for both Kubernetes itself and the external provisioner to support block volume functionality. Support for [block volume in Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support) was introduced in v1.9 and promoted to beta in Kubernetes 1.13\n\nWith the release of OpenEBS 0.7.0, users can create a block volume via the OpenEBS external-provisioner. With this provisioner, during volume creation, the user requests the PersistentVolumeClaim, which sets *volumeMode=”Block”* in the *PersistentVolumeClaimSpec*, binds it with *PersistentVolume* objects, and Block devices are eventually attached to the pods by including them in the volumes array of the *podSpec.*\n\nRegardless of the *volumeMode*, provisioner can set *FSType* into the plugin’s volumeSource, but the value will be ignored at the volume plugin side if volumeMode = Block. Leaving *volumeMode* blank is essentially the same as specifying *volumeMode = “Filesystem,”* which results in the traditional behavior.\n\nWhen using a raw block volume in your Pods, you must specify a *VolumeDevice *attributein the Container section of the *PodSpec* rather than a *VolumeMount*. *VolumeDevices* utilize *devicePaths* instead of *mountPaths*. Inside the container, applications will see a device at that path instead of a mounted file system.\n\n## How to Use Block Volume\n\nThere are a number of use cases where using a raw block device can be useful. For example, A user can use a raw block device for database applications such as MySQL to read data from and write the results to a disk that has a formatted filesystem to be displayed via the ***nginx*** web server.\n\nThe following sections detail some sample volume specifications and steps to dynamically provision a raw block volume and attach it to an application pod.\n\n## Creating a new raw block PVC\n\nHere, the user creates one raw block volume and another formatted filesystem based volume that dynamically creates PersistentVolumes(PV).\n\n1. ***Raw Block volume:***\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-block-pvc\n spec:\n volumeMode: Block\n storageClassName: openebs-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\n2. ***Filesystem based volume:***\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol-pvc\n spec:\n storageClassName: openebs-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\n## Using a raw block PVC in POD\n\nHere, the user creates an application pod whose containers consume both block and filesystem volumes. We must choose devicePath for the block device inside the Mysql container rather than the mountPath for the file system.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: my-db\n spec:\n volumes:\n - name: my-db-data\n persistentVolumeClaim:\n claimName: demo-block-pvc\n - name: my-nginx-data\n persistentVolumeClaim:\n claimName: demo-vol-pvc\n containers\n - name: mysql\n image: mysql\n volumeDevices: \n - name: my-db-data\n devicePath: /var/lib/mysql/data\n - name: nginx\n image: nginx\n ports:\n - containerPort: 80\n volumeMounts:\n - mountPath: /usr/share/nginx/html\n name: my-nginx-data \n readOnly: false\n\n## Conclusion\n\nA *block volume* is a volume that appears as a block device inside the container and allows low-level access to the storage without intermediate layers, as with file-system volumes. There are several advantages *of raw disk partitions*, including:\n\n- Block devices that are actually SCSI disks support the sending of SCSI commands to the device using Linux ioctls.\n- Faster I/O without an overhead UNIX file system, more synchronous I/O without UNIX file system buffering, etc.\n\nThanks to [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page).\n","notHasFeatureImage":false,"slug":"block-volume-support-for-openebs-volume"},{"id":53,"title":"Litmus - Journey of a storage e2e and chaos framework for Kubernetes","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"26-03-2019","tags":["Chaos Engineering","Continuous Integration","Kubernetes","Litmus","OpenEBS"],"excerpt":"The objective of the Litmus project is to help evaluate stateful workloads on Kubernetes via storage & infrastructure focused end-to-end (e2e) tests that use chaos engineering tools & principles","content":"\n**This article is part of a #ProjectUpdates series on Litmus. In this blog post, we will share the latest developments in the project, which was announced & open-sourced during the [KubeCon Europe 2018](/blog/litmus-release-a-chaos-monkey-on-your-kubernetes-stateful-workloads)**\n\nThe objective of the Litmus project is to help evaluate stateful workloads on Kubernetes via storage & infrastructure focused end-to-end (e2e) tests that use chaos engineering tools & principles, amongst other aids, with the test cases executed as kubernetes jobs. Over the last 10 months, there have been several improvements to the framework based on community feedback alongside an increase in the test library. The focus areas & improvements through this period, ongoing efforts & upcoming features have been summarised in the subsequent sections.\n\n## Increased application & use-case coverage\n\nWhile the initial focus of Litmus centered around creating a K8s native approach to e2e (via test containers deployed as jobs) & facilitating a simpler interface for users to define & generate tests via English-like sentences (using godog), subsequent feedback from the extremely vibrant & involved OpenEBS community highlighted the need for users to be able to pick test cases, i.e., **deployable test jobs** from an existing test library to enable quick turnaround time for stateful workload evaluation. As a result, the existing ansible-based utils were extended.\n\nLitmus now includes support for multiple popular stateful applications (Percona MySQL, MongoDB, Apache Cassandra, MongoDB, Minio, Crunchy PostgreSQL, NuoDB, Prometheus etc..,) whilst covering common app lifecycle scenarios such as deployment, load, reschedule, scale-up, upgrade & de-provision. Also included are test jobs to verify the resiliency of these workloads against intermittent and permanent failures to persistent storage & cluster infrastructure, with these failures induced via chaos tools.\n\n## Refactoring Litmus into a more “microservice-oriented” model\n\nOne of the principles in the microservices world involves compartmentalization of an application service into several pieces to enable greater flexibility & control in administration. When the Litmus project was started, the entire workflow of the test: right from application deployment, through test business logic to clean up was embedded into a single job. While this continues to be a viable model in some circumstances, managing multiple user inputs/permutations at each stage of the workflow meant handling this in a single job would make the test a complex *“monolith”* & affect debuggability.\n\n![Continuous Integration (CI) pipelines](https://blog.mayadata.io/hs-fs/hubfs/Blog%20Images-60.png?width=834&name=Blog%20Images-60.png)Continuous Integration (CI) pipelines\n\nAnother consideration was Litmus tests being used to drive Continuous Integration (CI) pipelines that aim to simulate user environments, where applications are typically soaked over a period of time undergoing various *day-1/day-2-n* ops while being resilient to component failures. This resulted in the creation of different categories of Litmus jobs (henceforth interchangeably referred to as Litmusbooks), which can be combined in a specific order to achieve an “e2e test” or “Litmus experiment”.\n\n**App Deployers/De-provisioners:** Litmusbooks that take various parameters such as namespace, type, labels, liveness (IO) probes etc., to deploy or clean up the aforementioned applications.\n\n**Load generators:** Generates both synthetic (fio, sysbench, dd) & app-specific (TPC-C, YCSB) IO load on the applications using custom containers simulating client behavior.\n\n**External liveness monitors:** Monitors accessibility to an application server. While the standard K8s liveness/IO probes perform pod restarts to restore app server health, these external monitors test the seamless continuity of “*end-applications*” during the course of an app-lifecycle operation or chaos.\n\n**App/Storage lifecycle “operators”:** Performs standard functional operations such as app/storage scale, backup/restore, reschedule, app/storage upgrades etc.., (should not be mistaken for actual “kubernetes operators”)\n\n**Chaos inducers:** Litmusbooks that induce failures on specified targets on the cluster, including application replicas, storage & cluster infrastructure (network, nodes, disks, memory, CPU)\n\n![Litmus Framework](/images/blog/litmus-framework.png)\n\n(***Litmus Framework***)\n\nTypically, the lifecycle functionality & chaos litmusbooks include steps for post-test health checks on the application and persistent storage components, including **data integrity** validation. All the above requirements necessitated the creation of multiple “*test facilitator containers*”(maintained [here](https://github.com/openebs/test-tools)) that implement python-based application clients and integrate other popular opensource chaos tools such as **pumba**, **chaostoolkit** & **chaoskube**.\n\nFor more details on chaos engineering using Litmus, read this [article](/blog/storage-chaos-engineering-with-litmus-an-overview?__hstc=216392137.a909c2d2cc44264c6d1aa717b549f14e.1580153178674.1580153178674.1580153178674.1&__hssc=216392137.1.1580153178674&__hsfp=3765904294).\n\n## Litmus e2e test result visualization\n\nOne of the important enablers for the adoption of any given test (suite) into a CI framework is the result mechanism to determine success/failure. During the said period, Litmus moved over from extracting results from test container logs to storing this information in a Kubernetes static custom resource (*“litmusresult”, which is updated by the test code at the beginning & end of test execution via a JSON merge patch operation*), with the resource name dynamically generated based on user tags. This resource also holds other test metadata such as *appType*, *testType*, *testStatus* etc., apart from the actual *verdict* (result). CI jobs that implement/run the litmusbooks can be configured to derive job status from these CRs.\n\n![cr-structure](/pubic/images/blog/cr-structure.png)A jinja template describing the result CR structure\n\n## Other notable enhancements\n\nThe project also saw some other organic changes which have resulted in its faster adoption into CI systems & increased efficiency of the tests. They have been briefly described below:\n\n**Infra Setup Playbooks**: Litmus now provides multiple ansible playbooks to aid quick and easy creation of Kubernetes clusters on different popular cloud platforms (GCP, AWS, GKE, AKS, EKS, Packet Cloud, On-premise vSphere VMs, OpenShift-on-AWS, OpenShift-on-vSphere) with minimal config inputs & also litmusbooks to setup the OpenEBS control plane on a K8s cluster.\n\n**Simplifying Litmus Setup**: The litmus pre-requisites are taken care of by a simple interactive bash script that sets up the RBAC, CRDs & config file updates.\n\n**K8s-Specific Enhancements**: These include small changes to make the litmusbooks more robust, such as:-\n\n- Usage of generateName API in the litmus jobs to ensure repeated execution doesn’t cause failures\n- Maximum Usage of labels & JSONPATH expressions in the kubectl commands used in the test code instead of bash utilities to guard against K8s cluster version & storage resource changes.\n- Increased use of pod ENV variables (user-specific & downward API) to control test inputs in a K8s-native way rather than pass them in a test-code specific form (ansible extra-vars)\n\n**Increased Modularization of Test Code**: Most of the test code is reused across the litmusbooks, due to which reusable tasks are managed as separate utils. These are further classified as *common*, *chaos* & *app/storage lifecycle* modules, with well-defined interfaces.\n\n**Litmus CI**: The Litmus (and test-tools) project now consists of its own Travis-based CI which builds respective runner images post-Ansible/Python lint/syntax check operations.\n\n**Optimized Test Images**: The litmus runner (go, ansible) as well the test facilitators images have been optimized with respect to the number of layers, size and binary versions used.\n\n## Case Study: OpenEBS CI Powered By Litmus\n\n![openebs-ci-powered-by-litmus](/images/blog/openebs-ci-powered-by-litmus.png)\n\n(***OpenEBS CI Powered By Litmus***)\n\nThe Continuous Integration framework for the OpenEBS project, i.e., the various components such as Maya, Node-Disk-Manager (control plane), Jiva, cStor (data plane) is built using **Gitlab**, with the e2e tests executed via **Litmusbooks** & supported by an Elasticsearch-Fluentd-Kibana (**EFK**) based logging framework. Post the build phase (where the above-mentioned components are built and images pushed to respective repositories) multiple gitlab e2e pipelines based on the Kubernetes cluster version are triggered, each of which includes the following litmus-powered stages to:\n\n- Create K8s cluster on the bare metal cloud platform Packet\n- Deploy the OpenEBS control plane\n- Deploy stateful applications\n- Run the app & storage lifecycle ops\n- Induce application, OpenEBS persistent storage & cluster infrastructure Chaos\n- De-provision the applications & OpenEBS control plane\n- Delete the K8s cluster resources\n\nThe **gitlab runner** scripts follow a defined pattern that involves the litmusbook preconditioning (user inputs), followed by litmusbook execution & monitoring of result custom resources, from which the success of gitlab job (and hence the stage) is derived.\n\n[openebs.ci](https://openebs.ci/) *provides a dashboard view of the build status for various commits to the openebs control plane and data plane projects, with detailed visibility into individual stage/job status & respective logs on Kibana. It also provides a real-time view of long-running production workloads using OpenEBS PVs which are maintained over multiple OpenEBS releases, with an option to induce random chaos on application/storage components & view the behavior. Litmus chaos modules are called internally to achieve this.*\n\nMore details on the composition & functioning of this CI framework is available in this [blog](/blog/a-primer-on-openebs-continuous-integration?__hstc=216392137.a909c2d2cc44264c6d1aa717b549f14e.1580153178674.1580153178674.1580153178674.1&__hssc=216392137.1.1580153178674&__hsfp=3765904294) post.\n\n## Ongoing Efforts & Upcoming Features In Litmus\n\nAt this point, the Litmus contributions are mainly focused towards increasing the application & storage life cycle tests, more specifically, **Day-2 operations** of standard stateful applications, while continuing to increase the **stability** of the current tests.\n\nSome of the near-term improvements & enhancements include:\n\n- Publish documentation for the Litmus project\n- Improvement of the Litmus Result CR to hold real-time status with increased metadata on failures, including the ability to capture application state.\n- Development of additional infrastructure & application-specific chaos utils\n\n## Conclusion\n\nThe improvements made to the Litmus project have been possible only due to the amazing support from the community both in terms of feedback, as well as contributions. So, a huge thank you & looking forward to more of the same!!\n","notHasFeatureImage":false,"slug":"litmus---journey-of-a-storage-e2e-and-chaos-framework-for-kubernetes"},{"id":54,"title":"(Storage) Chaos Engineering with Litmus - An Overview","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"20-03-2019","tags":["Chaos Engineering","Docker","Kubernetes","Litmus","OpenEBS"],"excerpt":"In this extremely insightful article, Evan Powell outlines various possible failures that could occur in a Kubernetes cluster running stateful workloads & the possible means to mitigate them..","content":"\n**This article is part of a #Concepts series on Kubernetes and Litmus. Here, we present an overview of why & how the Litmus project employs chaos techniques to test the resiliency of OpenEBS storage solutions.**\n\nIn this extremely insightful [article](/blog/ha-vs-dr-and-ha-for-your-db), [Evan Powell](https://medium.com/u/36584a5b84a) outlines various possible failures that could occur in a Kubernetes cluster running stateful workloads & the possible means to mitigate them, while advocating Chaos Engineering as a means to gauge system behavior, blast radius & recovery times. While there are some amazing chaos tools in the Kubernetes ecosystem, both opensource ([chaoskube](https://github.com/linki/chaoskube), [pumba](https://github.com/alexei-led/pumba), [chaostoolkit](https://github.com/chaostoolkit/chaostoolkit)etc.,) & lately, free solutions ([gremlin-free](https://www.gremlin.com/blog/introducing-gremlin-free/)) which can help induce specific types of failures on containers, pods, virtual machines and cloud instances, a need was identified to tie the act of chaos itself with **automated orchestration** (steady state condition checks) & **analysis** (data availability & integrity, application & storage deployments’ health etc.,).\n\nLitmus was [developed ](https://medium.com/@karthik.s_5236/litmus-journey-of-a-storage-e2e-and-chaos-framework-for-kubernetes-dc09a3904a24)in order to (amongst other needs) facilitate a Kubernetes-native & persistent storage, i.e., stateful workload-focused approach to chaos, with ability to self-determine the success of a controlled chaos experiment based on pre-defined exit criteria. In other words, a framework to construct & host a ready set of “deployable tests” that use [chaos engineering principles](https://principlesofchaos.org/) as the underlying business-logic. In order to achieve this, Litmus internally makes use of the aforementioned opensource chaos tools, combined with the power of kubectl, while also employing certain home-grown chaos techniques where necessary.\n\nThough Litmus has been designed to be easily extendable to work with different persistent storage solutions, the [OpenEBS ](https://openebs.ci/)community is the primary consumer of these tests. As a result, the majority of the utils/modules are chaos functions directed against the OpenEBS storage (persistent volume) resources (target pod, replica pod, storage pools etc.,) and their corresponding state checks. There are, however, app-specific and general infrastructure chaos tests too.\n\n**T**he following sections will briefly discuss the approaches and patterns used in the design of the different litmus chaos tests. The objective here is to introduce the larger community to methods being followed in this project and receive valuable feedback against them! The step-by-step illustration on running these tests, with details on impact/behavior with respect to different OpenEBS storage engines, will be covered in separate blog posts in the #HowDoI series.\n\n**The Litmus Job Template**\n\n***Note**: _To set the context (though it may already be known), a majority of the litmus tests are essentially ansible playbooks being executed from within a runner pod, which in turn is managed by the Kubernetes job controller._*\n\nOne of the requirements of the “deployable” litmus test or chaos experiment is that the *control variables* (typically, chaos params — considering that different applications, storage engines, PV component versions, and deployment models are evaluated against a particular chaos sequence) and *independent variables* (PV, application and K8s deployment params) should be defined in the test artifact, which, in this case, is a Kubernetes job specification. This is achieved by specifying them as pod/container environment (ENV) variables. In most cases, the final object (cluster resource) of chaos is derived from the application or persistent volume claim info passed in the job.\n\n![sample litmus chaos test](/images/blog/sample-litmus-chaos-test.png)\n\n(***Sample Litmus Chaos Test Variables/Tunables***)\n## Test Execution Workflow\n\nThe standard chaos test execution workflow is described in the schematic diagram shown below.\n\n![litmus chaos test execution](/images/blog/litmus-chaos-test-execution.png)\n\n(***Litmus Chaos Test Execution Workflow***)\n## Container Crash\n\nLitmus makes use of a [Pumba daemonset](https://github.com/openebs/litmus/blob/master/chaoslib/pumba/pumba_kube.yaml) to set up the chaos infrastructure, as this is necessary for the target and pumba containers to co-exist on a node so the latter can execute the Kill operation on the target. These pods are configured to run in --dry mode and have the host’s /var/run/docker.sock mounted as a volume, thereby allowing the test to launch custom chaos commands using the *pumba-cli* through kubectl exec operations.\n\nPumba “kills” a container by terminating the main process on the target container, and multiple termination signals (SIGKILL, SIGTERM, SIGSTOP)are supported. The object to be killed is either the app or PV target/replica/pool containers or their sidecars.\n\nA sample chaos util (essentially, a task file that is imported/called in the main chaos test playbook) for performing a container kill can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/pumba/pod_failure_by_sigkill.yaml).\n\n## Forced Pod Reschedule\n\nPod deletion, which is managed by higher level kubernetes controllers such as deployments and statefulsets, causes the pod to be rescheduled on a suitable node (same or different) in the cluster. Continuity of end application I/O and the time taken for reschedule/bring-up are the crucial post checks performed in these tests.\n\nLitmus uses a [chaoskube deployment](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/chaoskube.yaml) (setup with RBAC) configured in a default --dry-run mode, allowing the test to run a custom pod failure chaos command using the *chaoskube-cli* through kubectl exec operations. This is done by providing appropriate label and namespace filters to point to the desired target pods.\n\n*Most of the chaos tests confirm the success of chaos induction (which itself “can” fail) by observing an increment in the *resourceVersion* of the target deployments (component under test) as part of the post-chaos checks.*\n\nA sample chaos util for performing a forced pod reschedule using chaoskube can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/pod_failure_by_chaoskube.yaml).\n\n## Network Delay and Packet loss to Specific Storage Resources\n\nLitmus uses the Pumba daemonset (which internally makes use of [tc ](https://linux.die.net/man/8/tc)and [netem](http://man7.org/linux/man-pages/man8/tc-netem.8.html) tools on the target containers) and tc directly in some cases to generate *egress delays* and *percentage packet loss* on the specified targets for the desired duration. Pumba also has the capability to induce network delays using random distribution models (uniform, normal, Pareto, Paretonormal) and loss models (Bernoulli, Markov, Gilbert-Elliott).\n\nThese capabilities are used to simulate flaky iSCSI storage targets, lousy data-replication channels, and test data reconstruction post temporary component loss (such as storage replicas and pools).\n\nSample chaos utils for inducing network delays and packet loss can be viewed [here ](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/cstor_target_network_delay.yaml)and [here,](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml)[https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml)respectively.\n\n## Forced Reschedule via Taint-Based Evictions\n\nUnder certain specific node conditions/states, Kubernetes automatically taints the node with what is commonly referred to as “*eviction taints*”. These have a “*NoExecute*” taint effect. This means the pods scheduled and running on these nodes are immediately evicted and rescheduled onto other suitable nodes on the cluster. Amongst these taints, Kubernetes automatically adds a default toleration period *(tolerationSeconds)* of 300s for **node.kubernetes.io/unreachable** & **node.kubernetes.io/not-ready** to account for transient conditions. For other taints, such as **node.kubernetes.io/memory-pressure**, **node.kubernetes.io/disk-pressure,** etc.., the pods are evicted immediately (unless specifically tolerated in the deployment spec. Visit [kubernetes docs](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) for more details).\n\nThis scenario holds relevance in the case of stateful workloads, where an *en-masse* movement of the application and storage resources (which may include control plane components) can have potential data availability considerations. In other words, it is an effective HA test, not dissimilar to a graceful node loss.\n\nLitmus uses **kubectl** to simulate such node conditions by manually applying the taint on desired nodes (typically, the one which hosts the application and storage target pods) while monitoring end-application I/O. A sample chaos util for eviction by taints can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/kubectl/pod_evict_by_taint.yaml). Litmus simulates graceful node loss via another mechanism: **Cordon and Drain** (this also triggers reschedule of all the node’s pods), which is typically used in cluster maintenance workflows. This util is available [here](https://github.com/openebs/litmus/blob/master/chaoslib/kubectl/cordon_drain_node.yaml)**.**\n\n## Simulated Daemon Service Crashes\n\nLoss of node services (*kubelet*, *docker*) is one of the most common reasons for failed/non-starting pods and inaccessible application/storage services on the cluster. Litmus uses a [privileged daemonset](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/svc_chaos_kube.yml) that mounts the host’s root filesystem paths in order to execute systemctl commands on the desired node (again, typically the node running application and storage target pods).\n\nIn case of managed (GKE) or cloud-hosted (KOPS based AWS, for ex.) kubernetes clusters, these crucial daemon services often have corresponding health check/monitoring services that detect anomalies and attempt a restart to restore node health. The litmus chaos utils disable these monitoring services temporarily for the duration of the chaos to ensure consistent chaos rather than flaky state changes (this, it can be argued, is a case for a separate test!). After the chaos duration has elapsed, all affected services are restored, with post checks confirming cluster health.\n\nThese scenarios cause the ungraceful/forced reschedule workflows in Kubernetes to kick-in. They have significant implications on stateful workloads, with disk/volume unmount-detach and reattach-remount times in focus. As with the eviction by taint and drain scenarios, Litmus monitors end-application I/O and data integrity as part of the chaos experiment.\n\nSample utils for service chaos on GKE and AWS clusters can be viewed [here ](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/svc_chaos.yml)and [here](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/aws-service.yml)**,** respectively.\n\n## Resource Exhaustion on Kubernetes Nodes\n\nWhile we discussed simulating node conditions via eviction taints, it is also necessary to actually induce resource exhaustion such as *CPU burn*, *Memory Hog*, and *Filled Disks* to gauge real-time behavior. Litmus uses simple docker containers that run (a) a python program to keep appending large data to a growing string to [consume system memory](https://github.com/openebs/test-tools/tree/master/memleak), and (b) a forkbomb used to cause process multiplication resulting in [CPU burn](https://github.com/openebs/test-tools/tree/master/forkbomb).\n\n![system memory consumption](/images/blog/system-memory-consumption.png)(a) \n(***system memory consumption***)\n\n![cpu burn](/images/blog/cpu-burn.png)(b) \n(***CPU burn via forkbomb***)\n\nA [node-action daemonset](https://github.com/openebs/litmus/blob/master/apps/percona/chaos/node_freeze/node_freeze.yml) that mounts the host’s **/var/run** is used to execute docker commands that run the above containers with Litmus monitoring node status and end-application I/O. The sample chaos util for resource exhaustion can be found [here](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/node_freeze_chaos.yml)**.**[Disk/Capacity fill utils](https://github.com/openebs/litmus/tree/master/apps/fio/tests/pool_capacity), available in Litmus currently, are implemented within the storage volume and storage pool layers using **fio** tool.\n\n## Kubernetes Node (instance) Failures\n\nLitmus supports instance termination on AWS (KOPS based Kubernetes clusters) and Packet (Bare-metal cluster using kubeadm). Work is in progress to extend the similar capabilities to other platforms as well. The different impact is noticed when node failures are performed across platforms. For example, in the case of AWS, another instance with the same root disk is automatically brought up as the nodes are by default a part of the KOPS auto-scaling instance group. With Packet, however, the node remains shut down and can be subsequently powered-on. These behaviors expose different workflows of the underlying storage solution.\n\nFor AWS, Litmus uses [chaostoolkit templates ](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/random_node_terminate_aws.j2)that are executed in a [Kubernetes job](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/node-stop.yml)**,** which in turns runs a [custom container](https://github.com/openebs/test-tools/blob/master/chaostoolkit-aws/Dockerfile) using chaostoolkit binary and is equipped with the desired libraries. The AWS access keys and instance details are passed as test inputs from the Litmus test job. The util is capable of random instance selection in a given *Availability Zone*, as well as termination of a specific instance.\n\nFor Packet, Litmus uses the [Ansible Packet module](https://docs.ansible.com/ansible/latest/modules/packet_device_module.html#packet-device-module) **to power-off** and power-on the node instance, with the packet *API keys*, *project id*, *node name* and *chaos duration* passed as test inputs.\n\nThe AWS and Packet node failure chaos utils can be found [here ](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/chaostoolkit_util.yml)and [here](https://github.com/openebs/litmus/blob/master/chaoslib/packet/node_chaos.yml).\n\n## Disk Failures\n\nLitmus currently supports disk failure chaos utils for Kubernetes clusters built on an AWS cloud. Work is in progress to extend this to other platforms. The AWS CLI and [Boto Python SDK](https://boto3.readthedocs.io/) are used to force detach and reattach the EBS volumes used as the disk sources for the storage pools. As with other chaos tests, Litmus monitors the end-application I/O and data integrity over the course of the experiment.\n\nA Sample EBS disk failure util can be found [here](https://github.com/openebs/litmus/blob/master/chaoslib/aws_chaos/disk_failure.yml)**.**\n\n## Conclusion\n\nAt this point in Litmus’ journey, improving the automated analysis capabilities, increasing the chaos function coverage across components, creating greater support for common stateful applications, and enhancing the stability of existing chaos tests are the top priorities. However, as happy members of the chaos engineering community, we are always on the lookout for updates and new entrants in the ecosystem and any other ways we can contribute! We hope this article has been helpful in gaining a better understanding of how Litmus does chaos. As mentioned earlier, we always look forward to your feedback & comments!!\n","notHasFeatureImage":false,"slug":"(storage)-chaos-engineering-with-litmus---an-overview"},{"id":55,"title":"PostgreSQL anywhere — via Kubernetes with some help from OpenEBS and credativ engineering","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"18-03-2019","tags":["DevOps","Kubernetes","Postgres","Postgresql","Solutions"],"excerpt":"In this blog, we’d like to briefly cover how using cloud-native or “container attached” storage can help in the deployment and ongoing operations of PostgreSQL on Kubernetes.","content":"\nby [Murat Karslioglu](https://twitter.com/muratkarslioglu), [OpenEBS](https://openebs.io/) and [Adrian Vondendriesch](https://twitter.com/__discostu__), [credativ](https://www.credativ.com)\n\n### Introduction\n\nIf you are already running Kubernetes on some form of cloud whether on-premises or as a service, you understand the ease-of-use, scalability and monitoring benefits of Kubernetes — and you may well be looking at how to apply those benefits to the operation of your databases.\n\nPostgreSQL remains a preferred relational database, and although setting up a highly available Postgres cluster from scratch might be challenging at first, we are seeing patterns emerging that allow PostgreSQL to run as a first class citizen within Kubernetes, improving availability, reducing management time and overhead, and limiting cloud or data center lock-in.\n\nThere are many ways to run high availability with PostgreSQL; for a list, see the [PostgreSQL Documentation](https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling). Some common cloud-native Postgres cluster deployment projects include [Crunchy Data](https://www.crunchydata.com/)’s, [Sorint.lab](https://www.sorint.it/)’s [Stolon](https://github.com/sorintlab/stolon) and [Zalando](https://jobs.zalando.com/tech/)’s [Patroni](https://github.com/zalando/patroni)/[Spilo](https://github.com/zalando/spilo). Thus far we are seeing Zalando’s operator as a preferred solution in part because it seems to be simpler to understand and we’ve seen it operate well.\n\nSome quick background on your authors:\n\n- OpenEBS is a broadly deployed OpenSource storage and storage management project sponsored by MayaData.\n- [credativ](https://www.credativ.com) is a leading open source support and engineering company with particular depth in PostgreSQL.\n\n> **In this blog, we’d like to briefly cover how using cloud-native or “container attached” storage can help in the deployment and ongoing operations of PostgreSQL on Kubernetes. This is the first of a series of blogs we are considering — this one focuses more on *why* users are adopting this pattern and future ones will dive more into the specifics of *how* they are doing so.**\n\n> **At the end you can see how to use a Storage Class and a preferred operator to deploy PostgreSQL with OpenEBS underlying**\n\nIf you are curious about what container attached storage of CAS is , you can read more from the Cloud Native Computing Foundation (CNCF) here:\n\n[https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/)\n\nConceptually, you can think of CAS as the decomposition of previously monolithic storage software into containerized microservices that run on Kubernetes. This provides the same advantages of running Kubernetes to the storage and data management layer as well. Like Kubernetes, OpenEBS runs anywhere, so the same advantages below apply whether it is used on-premises or on any of the hosted Kubernetes services.\n\n**PostgreSQL plus OpenEBS**\n\n![Postgres Operator using OpenEBS](/images/blog/postgres-operator-using-openebs.png)\n(***Postgres-Operator using OpenEBS as the Storage Provider***)\n\nWe have seen joint users adopting OpenEBS as a substrate to PostgreSQL for a variety of reasons. Here are a few of the reasons that stand out:\n\n**Consistency in underlying disk or cloud volume management.**\n\nOne of the most annoying things about setting up a system to run PostgreSQL, even if it is on Kubernetes, is configuring the underlying disks and storage systems as needed. With a solution such as OpenEBS, you specify how you want the underlying systems configured via storage classes. OpenEBS, with the help of Kubernetes, ensures that the system delivers the required storage capacity and is configured as you need. An example of such a storage class is shared below. This automation can remove a source of human error and annoyance.\n\n**Thin provisioning and on-demand expansion.**\n\nNow that you have turned over to OpenEBS and the provisioning and management of the underlying storage hardware and services, you simply tell it the amount of storage you need for your PostgreSQL and everything will work, right? In reality, knowing how much data your PostgreSQL instance or instances will consume is difficult and arguably impossible as it is beyond your control.\n\nOpenEBS can also help here as it supports both thin provisioning and on-the-fly pool expansion. The thin provisioning allows you to claim more space than you actually can provision, allowing your PostgreSQL to scale in space usage without interruption. This is done by allowing the addition of more storage to the running system without the need to stop the database.\n\nThin provisioning, however, is not a good idea if there is no on-the-fly expansion of the underlying capacity for obvious reasons: as the PostgreSQL expands, you want to ensure it can claim space as needed. Otherwise, at some point you will have to interrupt operations and again perform manual tasks. OpenEBS helps here as well — if configured to do so, it can expand its underlying pools, whether these are of physical disks, underlying storage systems, or storage services from a cloud. The capacity of the pool can be expanded on demand by simply adding more disks to the cStor pool.\n\nThe cStor architecture also supports the resizing of a provisioned volume on the fly, which will be fully automated as of OpenEBS 0.9. With these enhancements, volumes and underlying pools will be able to scale automatically on any cloud providing K8s support.\n\nIn addition to reducing the risk and hassle of manual operations, the combination of thin provisioning and on-demand scaling can reduce costs. This is the case because you don’t over-provision capacity to achieve performance, which reduces unnecessary cloud service spending and can increase average utilization of hardware usage as well.\n\n**Disaster recovery and migration.**\n\nWith a solution like OpenEBS, your storage classes can also include backup schedules. These can be easily managed via Kubectl, which is free to use. Again, these storage classes can be applied on a per-container basis, which offers quite a bit of granularity and control by each team running their PostgreSQL.\n\nAdditionally, we are working together to increase integration with PostgreSQL to this per-snapshot based workload, per container backup capability, called DMaaS by MayaData and OpenEBS. With this additional integration, an option will be added to the storage classes and to OpenEBS that flushes active transactions before taking the snapshot. The additional integration of storage snapshots in conjunction with Write Ahead Log (WAL) archiving will provide additional PITR functionality. DMaaS leverages the open-source Velero from Heptio and marries it to the COW based capabilities of the cStor OpenEBS engine to deliver highly-efficient backups and migrations.\n\nWith DMaaS, backups that are taken to one location can be recovered from another. This can be very useful for a variety of use cases, including the use of relatively ephemeral clusters as a part of a rolling upgrade of an environment. Additionally, the same capability can be used to move workloads from one Kubernetes environment to another, thereby reducing lock-in.\n\n**Snapshots and clones for development and troubleshooting.**\n\nDBAs have long been using snapshots and clones to assist in troubleshooting and allow teams to develop and test against a read-only copy of production data. For example, with OpenEBS you can easily use Kubernetes to invoke a snapshot, then promote that snapshot to a clone and spawn a container from that clone. You now have the ability to do anything you want with that container and the contained dataset, and can destroy it when you are finished.\n\nOne use case that clones can support is improved reporting. For example, let’s say you use computationally expensive analytical queries and build roll-up queries for monthly reports. With OpenEBS, it is simple to clone the underlying OLTP system, therefore allowing you to work on a static copy of your database and remove load from your production DBs. This ensures you have a verifiable source of information for those reports.\n\n**Running Postgres-Operator on OpenEBS**\n\n![PostgreSQL with OpenEBS persistent volumes](/images/blog/postgres-operator-on-openebs.png)(***PostgreSQL with OpenEBS persistent volumes***)\n\n**Software Requirements**\n\n- [Postgres-Operator](https://github.com/zalando/postgres-operator) (for cluster deployment)\n- [Docker](https://docs.docker.com/install/)\n- Kubernetes 1.9+\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- [OpenEBS](https://github.com/openebs/openebs)\n\n**Installing OpenEBS**\n\nIf OpenEBS is not installed in your K8s cluster, you can do it [here](/docs/user-guides/installation?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025).\n\n**Configure cStor Pool**\n\n1. If cStor Pool is not configured in your OpenEBS cluster, follow the steps presented [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025). As PostgreSQL is a StatefulSet application, it requires a single storage replication factor. If you prefer additional redundancy, you can always increase the replica count to 3.\nDuring cStor Pool creation, make sure that the maxPools parameter is set to >=3. If a cStor pool is already configured, move on to the next step. A sample YAML named openebs-config.yaml can be used for configuring cStor Pool and is provided in the Configuration details below.\n\n**openebs-config.yaml**\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE — Appropriate disks need to be fetched using `kubectl get disks`\n #\n # `Disk` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n # Replace the following with actual disk CRs from your cluster `kubectl get disks`\n # Uncomment the below lines after updating the actual disk names.\n disks:\n diskList:\n # Replace the following with actual disk CRs from your cluster from `kubectl get disks`\n # — disk-184d99015253054c48c4aa3f17d137b1\n # — disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # — disk-806d3e77dd2e38f188fdaf9c46020bdc\n # — disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # — disk-bad1863742ce905e67978d082a721d61\n # — disk-d172a48ad8b0fb536b9984609b7ee653\n — -\n\n**Create the Storage Class**\n\n1. You must configure a StorageClass to provision a cStor volume on a cStor pool. In this solution, we use a StorageClass to consume the cStor Pool. This is created using external disks attached on the Nodes. The storage pool is created using the steps provided in the [Configure StoragePool](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025) section. In this solution, PostgreSQL is a deployment. Because this requires replication at the storage level, the cStor volume replicaCount is 3. A sample YAML named openebs-sc-pg.yaml used to consume the cStor pool with a cStorVolume Replica count of 3 is provided in the configuration details below.\n\n**openebs-sc-pg.yaml**\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-postgres\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"3\" \n provisioner: openebs.io/provisioner-iscsi\n reclaimPolicy: Delete\n ---\n\n**Launch and Test Postgres Operator**\n\n1. Clone Zalando’s Postgres Operator.\n\n git clone https://github.com/zalando/postgres-operator.git\n cd postgres-operator\n\n**Use the OpenEBS Storage Class**\n\n1. Edit the manifest file and add openebs-postgres as the storage class.\n\n nano manifests/minimal-postgres-manifest.yaml\n\nAfter adding the storage class, it should look similar to the example below:\n\n apiVersion: \"acid.zalan.do/v1\"\n kind: postgresql\n metadata:\n name: acid-minimal-cluster\n namespace: default\n spec:\n teamId: \"ACID\"\n volume:\n size: 1Gi\n storageClass: openebs-postgres\n numberOfInstances: 2\n users:\n # database owner\n zalando:\n - superuser\n - createdb\n # role for application foo\n foo_user: []\n #databases: name->owner\n databases:\n foo: zalando\n postgresql:\n version: \"10\"\n parameters:\n shared_buffers: \"32MB\"\n max_connections: \"10\"\n log_statement: \"all\"\n\n**Start the Operator**\n\n1. Run the following command to start the operator:\n\n kubectl create -f manifests/configmap.yaml # configuration\n kubectl create -f manifests/operator-service-account-rbac.yaml # identity and permissions\n kubectl create -f manifests/postgres-operator.yaml # deployment\n\n**Create a Postgres Cluster on OpenEBS (Optional)**\n\nThe operator can run in a namespace other than default. This may be done to use the test namespace, for example. Run the following before deploying the operator’s manifests:\n\n kubectl create namespace test\n kubectl config set-context $(kubectl config current-context) — namespace=test\n\n1. Run the command below to deploy from the example manifest:\n\n kubectl create -f manifests/minimal-postgres-manifest.yaml\n\n2. It only takes a few seconds to get the persistent volume (PV) for the *pgdata-acid-minimal-cluster-0* up and running. Check the PVs created by the operator using the k*ubectl get pv* command:\n\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-8852ceef-48fe-11e9–9897–06b524f7f6ea 1Gi RWO Delete Bound default/pgdata-acid-minimal-cluster-0 openebs-postgres 8m44s\n pvc-bfdf7ebe-48fe-11e9–9897–06b524f7f6ea 1Gi RWO Delete Bound default/pgdata-acid-minimal-cluster-1 openebs-postgres 7m14s\n\n**Connect to the Postgres Master and Test**\n\n1. If not installed previously, install psql client using this command:\n\n sudo apt-get install postgresql-client\n\n2. Run the command below and note the hostname and host port.\n\n kubectl get service — namespace default |grep acid-minimal-cluster\n\n3. Next, run the following commands to connect to your PostgreSQL DB and test. Replace the *[HostPort]* below with the port number from the output of the above command:\n\n export PGHOST=$(kubectl get svc -n default -l application=spilo,spilo-role=master -o jsonpath=\"{.items[0].spec.clusterIP}\")\n export PGPORT=[HostPort]\n export PGPASSWORD=$(kubectl get secret -n default postgres.acid-minimal-cluster.credentials -o ‘jsonpath={.data.password}’ | base64 -d)\n psql -U postgres -c ‘create table foo (id int)’\n\nCongratulations! You now have the Postgres-Operator and your first test database operating with the help of cloud-native OpenEBS storage.\n\n**Partnership and Future Direction**\n\nAs this blog indicates, the teams at MayaData/OpenEBS and credativ are increasingly working together to offer more benefits for organizations running PostgreSQL and other stateful workloads. In future blogs, we’ll provide more hands-on tips.\n\nWe are certainly looking for feedback and suggestions on the strategic direction of this collaboration. Please provide feedback below or find us on [Twitter](https://twitter.com/openebs) or the OpenEBS slack community: [http://slack.openebs.io](http://slack.openebs.io/?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025)\n","notHasFeatureImage":false,"slug":"postgresql-anywhere — via-kubernetes-with-some-help-from-openebs-and-credativ-engineering"},{"id":56,"title":"How to deploy NuoDB on Kubernetes with OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"14-03-2019","tags":["DevOps","Kubernetes","Nuodb","Solutions","OpenEBS"],"excerpt":"NuoDB is a container-native adaptive-scale distributed SQL database designed with distributed application deployment challenges in mind.","content":"\n**NuoDB** is a container-native adaptive-scale distributed SQL database designed with distributed application deployment challenges in mind. The NuoDB database is ANSI SQL compliant and seen in a new category called “NewSQL.” If you are not familiar, what is different with NewSQL solutions is explained in a 2016 research paper [here](https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs/07_forum_Pavlo.pdf).\nSome of the solutions in this category are [Amazon Aurora](https://en.wikipedia.org/wiki/Amazon_Aurora), [Google Spanner](https://en.wikipedia.org/wiki/Google_Spanner), TiDB, [CockroachDB](https://en.wikipedia.org/wiki/Cockroach_Labs), [Altibase](https://en.wikipedia.org/wiki/Altibase), [Apache Ignite](https://en.wikipedia.org/wiki/Apache_Ignite), [GridGain](https://en.wikipedia.org/wiki/GridGain_Systems), [Clustrix](https://en.wikipedia.org/wiki/Clustrix), [VoltDB](https://en.wikipedia.org/wiki/VoltDB), [MemSQL](https://en.wikipedia.org/wiki/MemSQL), [NuoDB](https://en.wikipedia.org/wiki/NuoDB), [HarperDB](https://en.wikipedia.org/w/index.php?title=HarperDB&action=edit&redlink=1), and [Trafodion](https://en.wikipedia.org/wiki/Trafodion)\n\nNuoDB has a distributed object architecture that works in the cloud, which means that in order to scale-out the database, a new node can be added and database runs faster. That’s why it represents the perfect example of a new-gen cloud-native architecture and a solution for the containerized applications running on Kubernetes.\n\nNuoDB provides all the properties of ACID-compliant transactions and standard relational SQL language support. It’s designed from the start as a distributed system that scales the way a cloud-native service is expected to scale, providing **continuous availability (CA)** and resiliency with no single points of failure. Different from traditional shared-disk or shared-nothing architectures, NuoDB presents a new kind of peer-to-peer, on-demand independence that yields CA, low-latency, and a deployment model that is easy to manage. The chart below is a simple comparison and decision chart from the [*NuoDB website*](https://www.nuodb.com/product/database-comparison).\n\n![Choosing the right database](https://cdn-images-1.medium.com/max/800/1*lQzxj0rx5WUBhinV2oaMsQ.jpeg)\n\n### Quick Notes on the NuoDB Architecture\n\n**NewSQL** systems are designed to operate in a distributed cluster of shared-nothing nodes, in which each node owns a subset of the data.\n\nAlthough it appears as a single, logical SQL database to the application, it has two-layers that retain strict transactional consistency. It can even be deployed across multiple availability zones (even on different clouds!) and is optimized for in-memory speeds, continuous availability, and adaptive scale-out that adjusts to the application needs.\n\n![NuoDB Architecture](/images/blog/nuodb-architecture.png) \n(***NuoDB Architecture: TEs (Top), SMs (Bottom)***)\n\nTransaction Engines (TEs): The TE layer is used for (ACID) SQL and caching, made up of in-memory process nodes that coordinate with each other and the SM layer.\n\nStorage Managers (SMs): The SM layer is used for storage and consists of process nodes that have both in-memory and on-disk storage components. SMs provide on-disk data durability guarantees, and multiple SMs can be used to increase data redundancy.\n\nBoth TE and SM node types can be easily added or removed to align with user needs.\n\n### Why Use OpenEBS for NuoDB?\n\nRunning applications on traditional virtualized systems or bare metal are different from running them on Kubernetes. NuoDB is a horizontally scalable database that offers scale up in performance on the fly from the underlying storage system. **Container Attached Storage** (CAS) solutions like OpenEBS are a perfect choice to run NuoDB on Kubernetes, both on-premise (such as OpenShift) and through the cloud because of the following benefits:\n\n- Local disks or cloud disks are managed seamlessly and in the same way by OpenEBS.\n- Persistent volumes (PVs) can be made available to NuoDB from a shared pool of disks. You can run multiple stateful databases out of the same physical disks present on the Kubernetes worker nodes, simplifying operations, increasing density, and decreasing costs.\n- Large size PVs can be provisioned for NuoDB. OpenEBS supports volumes up to petabytes in size.\n- You can increase the size of PVs as needed. With OpenEBS it is possible to start with small capacities and add disks as needed on the fly. Sometimes NuoDB instances are scaled up because of the capacity on the nodes. With OpenEBS persistent volumes, capacity can be thin provisioned, and disks can be added to OpenEBS on the fly without disruption.\n- NuoDB backups can be taken at the storage level and stored in S3. Using the backup/restore capabilities of OpenEBS, NuoDB instances can be moved across clouds or Kubernetes clusters seamlessly. There is no vendor lock-in problem when OpenEBS is used as the underlying storage.\n- Last, but not least — OpenEBS is 100% open source and in userspace.\n\n![production deployment of NuoDB](/images/blog/production-deployment-of-nuodb.png) (***Typical production deployment of NuoDB using OpenEBS.***)\n\nLet’s deploy our first NuoDB instance on our existing small, three-node OpenShift K8s cluster with OpenEBS.\n\n**Note:** NuoDB Community Edition supports only 1 SM and 3 TE configurations. To increase the number of SMs and TEs, the Enterprise edition should be used instead.\n\n### Requirements\n\n### Software\n\n- RHEL (master and node servers) 7.0 and above\n- [OpenShift](https://containerized.me/how-to-install-openebs-on-openshift/) 3.9 or above\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- [OpenEBS](https://github.com/openebs/openebs) v0.8.1 or above\n- NuoDB 3.4 Enterprise or Community Edition\n\n### Hardware\n\n- 3 nodes (bare metal or cloud)\n- 16 vCPUs\n- 32GB RAM\n\n### How to Install NuoDB\n\nIf you don’t have an OpenShift cluster, you can follow my previous blog [here](https://containerized.me/how-to-install-openebs-on-openshift/) to get both OpenShift and cloud-native storage OpenEBS up and running on Kubernetes.\n\nThe *NuoDB CE OpenShift* template is available with two options: using ephemeral storage or using persistent storage. With ephemeral storage, if the Admin Service pod or Storage Manager (SM) pod are stopped or deleted, the database will be removed with the pod as expected. That is where **OpenEBS** comes into play. When running NuoDB on OpenEBS, even if the pod is stopped or deleted, the DB state will be preserved and made available again on the automatic pod restart.\n\n### Disable Transparent Huge Pages (THP)\n\nAdditionally, on your OpenShift nodes run the command below to ensure Transparent Huge Pages (THP) is disabled. NuoDB will not run on a Linux machine with THP enabled.\n\n $ cat /sys/kernel/mm/transparent_hugepage/enabled\n\nIf this returns *|always| madvise never*, THP is enabled and you must disable it by adding the *transparent_hugepage=never* kernel parameter option to the grub2 configuration file. Append or change the “*transparent_hugepage=never*” kernel parameter on the *GRUB_CMDLINE_LINUX* option in the /etc/default/grub file similar to the below:\n\n $ vi /etc/default/grub GRUB_TIMEOUT=5 GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT=”console” GRUB_CMDLINE_LINUX=”nomodeset crashkernel=auto rd.lvm.lv=vg_os/lv_root rd.lvm.lv=vg_os/lv_swap rhgb quiet transparent_hugepage=never” GRUB_DISABLE_RECOVERY=”true”\n\n### Create a Project\n\nCreate a project named “nuodb” by clicking the OpenShift “Create Project” button. If desired, you can also add a template display name of “NuoDB on OpenEBS.”\n\n### Create an Image Pull Secret\n\nIf you want to use the official Red Hat Container Registry, then you need to create an image pull secret and import the image from the Red Hat Container Catalog (RHCC).\n\nTo create an image, use pull secret which allows the template to pull the NuoDB CE container image from the RHCC. From the OpenShift left bar menu, click “Resources,” then “Secrets,” and “Create Secret.”\n\nEnter these values:\n\n Secret Type = Image Secret\n Secret Name = pull-secret\n Authentication Type = Image RegistryCredentials \n Image Registry Server Address = registry.connect.redhat.com\n Username = (your RH login)\n Password = (your RH password)\n Email = (your email address)\n Link secret to a service account = (check this box)\n Service Account = Default\n\nThis should look similar to the screenshot below:\n\n![Create an Image Pull Secret](/images/blog/create-an-image-pull-secret.png)\n\nYou can find the RHCC instructions [here](https://access.redhat.com/containers/?tab=images&platform=openshift#/registry.connect.redhat.com/nuodb/nuodb-ce) if needed.\nAs an alternative, you can use the Docker registry images (not suggested for production).\n\n### Label the Nodes\n\nLabel at least three servers using the oc label command:\n\n $ oc label node nuodb.com/zone=nuodb\n\n### Configure cStor Pool\n\nAfter OpenEBS is installed, cStor pool must be configured. If cStor Pool is not already configured in your OpenEBS cluster, this can be done following the instructions [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools). A sample YAML named openebs-config.yaml used for configuring cStor Pool is provided in the Configuration details below. During cStor Pool creation, make sure that the maxPools parameter is set to >=3.\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get disks`\n #\n # `Disk` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n # Replace the following with actual disk CRs from your cluster `kubectl get disks`\n # Uncomment the below lines after updating the actual disk names.\n disks:\n diskList:\n # Replace the following with actual disk CRs from your cluster from `kubectl get disks`\n # - disk-184d99015253054c48c4aa3f17d137b1\n # - disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # - disk-806d3e77dd2e38f188fdaf9c46020bdc\n # - disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # - disk-bad1863742ce905e67978d082a721d61\n # - disk-d172a48ad8b0fb536b9984609b7ee653\n ---\n\n### Create the Storage Class\n\nYou must configure a StorageClass to provision a cStor volume on a given cStor pool. StorageClass is the interface through which most of the OpenEBS storage policies are defined. I will be using a StorageClass to consume the cStor Pool that is created using external disks attached on the Nodes. Since NuoDB is a statefulset application, it requires only a single storage replica. Therefore, the cStor volume replicaCount is =1. A sample YAML named openebs-nuodb.yaml used to consume cStor pool with cStoveVolume Replica count as 1 is provided in the configuration details below.\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-nuodb\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"1\" \n provisioner: openebs.io/provisioner-iscsi\n reclaimPolicy: Delete\n ---\n\n### Import the NuoDB CE Template\n\nImport the NuoDB CE template *ce-template-persistent.yaml* into OpenShift by navigating to the “Overview” Tab, clicking “Import YAML/JSON,” and running the import.\n\n![Import the NuoDB CE Template](/images/blog/import-the-nuodb-ce-template.png)\n(***Import YAML Template***)\n\nFollow the installation prompts and change the “SM persistent storage class name” and “Admin persistent storage class name” to *openebs-nuodb*.\n\n![Set Persistent Storage Class Name](/images/blog/set-persistent-storage-class-name.png) \n(***Set Persistent Storage Class Name***)\n\nClick “Create,” and shortly after you will see one pod each for the Administrative Service (Admin), Storage Manager (SM), Transaction Engine (TE) and Insights processes started under the *nuodb* namespace with volumes provided by OpenEBS.\n\n![nuoDB Successfully Deployed](/images/blog/nuodb-successfully-deployed.png) \n(***NuoDB Successfully Deployed***)\n\nIf you’d like to see or discuss anything not covered here, feel free to comment on my blog or contact me via Twitter @muratkarslioglu.\n\n---\n\n*Originally published at *[*containerized.me*](https://containerized.me/how-to-deploy-nuodb-on-kubernetes/)* on March 14, 2019.*\n","notHasFeatureImage":false,"slug":"how-to-deploy-nuodb-on-kubernetes-with-openebs"},{"id":57,"title":"Inspired by SCale17x — a short history of data storage","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"11-03-2019","tags":["DevOps","OpenEBS","Kubernetes","Open Source"],"excerpt":"Last weekend I had the pleasure of attending SCale 17x in Pasadena, just outside Los Angeles. Whether it was Robert Treat’s excellent talk on logical replication for PostgreSQL or refreshing my knowledge of Terraform","content":"\nLast weekend I had the pleasure of attending SCale 17x in Pasadena, just outside Los Angeles. Whether it was Robert Treat’s excellent talk on [logical replication for PostgreSQL](https://www.socallinuxexpo.org/scale/17x/presentations/postgres-logical-replication-lets-do-it-live) or refreshing my knowledge of Terraform thanks to the [dynamic and even funny instruction](https://www.socallinuxexpo.org/scale/17x/presentations/terraform-50-minutes) of [Rami Al-Ghanmi](https://twitter.com/alghanmi),  it was a great weekend filled with high-quality discussions. Kudos to Ilan and the rest of the all-volunteer organizing staff. Most importantly, it was a chance to run into friends, gather the MayaData/OpenEBS team together, support our CTO Jeffry as he told the story about [CAS (a great, deeper dive into the why and how of CAS)](https://www.socallinuxexpo.org/scale/17x/presentations/container-attached-storage-cas-openebs) and meet countless engineers, especially a bunch in the PostgreSQL, Percona, MariaDB, and MySQL communities.\n\nPerhaps my favorite moment was when I introduced myself during the DevOps LA taco night and the immediate reaction from a lead developer at a large analytics company was:\n\n“Oh holy #hit, OpenEBS and CAS — we need this stuff; working on it now as we move onto Kubernetes!”\n\nAs the conversation continued, a couple of questions emerged:\n\n1. How mature is it?\n2. Where did the idea come from?\n\nRegarding #1 — *maturity* — I think the growth of references on the community and our work at [www.openEBS.ci](http://www.openebs.ci/) to test every commit to master in public across many environments with many workloads both speak to the ongoing polishing and maturity of OpenEBS. I won’t dig too far into that subject, but I will say it is a lot more mature than many hacked together attempts at simply using a local PV. However, it is still <1.x.\n\nYou can read more about our efforts to mature OpenEBS and its docs and related management software in our founder’s recent blog post on the 0.8.1 release:\n\n[https://openebs.io/blog/openebs-releases-081-with-stability-fixes-and-improved-documentation](/blog/openebs-releases-081-with-stability-fixes-and-improved-documentation)\n\nMaturity and polishing is our top area of focus, and it is coming along quickly with the help of additional users and our dedicated, well-experienced engineering team. The next time you see them [on slack](https://openebs-community.slack.com/), please say thank you; the effort level by all involved is phenomenal.\n\nRegarding #2 — *where did the idea come from* — when trying to answer that question, I often roll the tape back a bit to see where storage has come from.\n\nWe put together a few slides that offer a light-hearted approach to answering this question.\n\n**The Era of Slow Disk Drives and Scale-up Architectures**\n\n![Slow Disk Drives and Scale-up Architectures](https://cdn-images-1.medium.com/max/800/0*6SPflCuTGC2rWXmH)\n\nSo, rolling the tape back to the 2000s, there were at least a few major differences as the slide expresses.\n\n1. At the bottom were slow disk drives; striping was required for adequate performance.\n2. Apps were not built for failure — users expected the storage to be bulletproof; on the other hand, once you got the App working, it wasn’t going anywhere, it was a monolith that could be tightly bound to the hardware and to the network attached storage.\n3. Along those lines, your apps were definitely not built to run on a public cloud service. They ran on a box, in your data center.\n\nBack then, it all sort of made sense. You needed software and systems that could combine many servers and disks into a highly resilient system to make sure your monolith continued to operate.\n\n**What has changed and how is that changing the data layer?**\n\n![Flash and NVMe extremely fast. Striping slows them down. DAS beats storage](https://cdn-images-1.medium.com/max/800/0*ZJ8CaKQdxEesL7eE)\n\nStorage was disrupted by flash. But now we are done with that, right? Well, not really…. Today’s scale-out storage systems almost always impose massive latency versus underlying, extremely fast NVMe connected non-volatile systems. As a result, and from a desire to avoid the complexity of running a distributed system like Cassandra on top of a distributed storage system (often with the distributed Kubernetes system in between), DAS has beaten all existing storage for many workloads.\n\nIn short, **the very reason that we started striping across nodes — to add resilience and performance to slow disks — no longer applies.**\n\nSo, what else?\n\n![75x increase in the number of workloads. 95% decrease in size & duration. Ephemeral](https://cdn-images-1.medium.com/max/800/0*VzhXdDLpxtERp1Sk)\n\nPerhaps most importantly, the workloads — the way DBs are built and run — has changed. They are of much shorter duration, typically much smaller, and they move around! This is not your father’s monolith!\n\n![You pay clouds for storage. Data gravity -> ^ spending, less agility. Lock-in](https://cdn-images-1.medium.com/max/800/0*Yuk0vAdoZe4qp24s)\n\nThen you run it on the cloud, which gives you and your teams incredible agility. However, you now introduce dependencies because each cloud’s storage differs. So, if you are not careful, before you know it you are locked-in. Even for some of our users listed here, ^^ their AWS bills are growing quickly. And, again, your teams are less agile because there has been a collective decision to grab some opaque service such as a stripped down and standardized version of open source MySQL or Redis from another company.\n\nTo sum things up, the world has changed in many ways! Therefore, storage has changed too, right? Well, not really. Storage is hard. It is far easier through marketing and open source influence peddling to get your 20-year-old project labelled “cloud native” than it is to actually start an entirely new storage system that runs anywhere.\n\n![Run anywhere with dynamic provisioning. Don't stripe across nodes. Reduce SPOF](https://cdn-images-1.medium.com/max/800/0*vNlv_UcOVV5Y456W)\n\nTo begin at the bottom of the stack, you not only have to accommodate the presence of extremely fast storage hardware, you also have to accommodate running *on clouds* themselves. How can you do this? And how do you do so in a way that does not itself introduce another single point of failure?\n\nThe approach we take at OpenEBS is to run our storage capabilities in a decoupled way so that the controller — that serves the actual workloads — is itself stateless and easily rescheduled. Also, it runs in the user space in containers so that it really can run anywhere. It then coordinates reads and writes to logically underlying data containers.\n\nWe also keep the data local and utilize intelligence in our control plane to leverage Kubernetes to optimize data placement of primary data stores and replicas. There is much more I could discuss here, but I’ll leave it at this high level for now.\n\n![Every workload has its own data storage. Every team as well. Quick & agile](https://cdn-images-1.medium.com/max/800/0*ZFFma1SdaG9GaiAS)\n\nSecondly, instead of having a one size fits all storage system that is limited in its ability to make trade-offs for each of the potentially thousands of workloads, each workload has its own storage system with OpenEBS. The specific needs of those workloads and of the “two pizza” teams that build and run the workloads are each served by their own OpenEBS. This limits the need for cross-functional team meetings in which members argue about how they want the storage to behave. They simply invoke their particular flavor via a Helm chart and, with the help of the right storage classes, away they go!\n\n![Run Dbs w/o any headache. Achieve data agility and cost savings. No lock-in. Unicorn](https://cdn-images-1.medium.com/max/800/0*w9bt1flsblASWqcG)\n\nBecause you can customize OpenEBS to run however you want it and it abstracts away the underlying differences from storage vendors or storage services, you are now less locked-in than before.\n\nPlus, with OpenEBS and MayaData we are providing per workload back-up and recovery and entire workload data migration as well. Common use cases include disaster recovery of stateful workloads, where you cannot, of course, have the workload itself doing all its own replication if the disaster you are protecting against is a catastrophic situation on that workload. Other common use cases include moving workloads from one cloud to another to save funds or possibly moving workloads closer to other cloud services.\n\n**What’s next?**\n\nNo look back to history would be complete without a quick look at the not-too-distant future as well.\n\nIn our case, we are adding additional performance tuning capabilities to the per workload container attached storage of OpenEBS. This will allow you to more easily tune components of a database, for example.\n\nAdditionally, we are working closely with a variety of workload providers to improve the ongoing testing and, yes, maturation of DBs + OpenEBS on various clouds and on premises deployments. We are increasingly getting help in building and operating [OpenEBS.ci](http://openebs.ci/) for example. The idea is to make it much easier to run your own databases without the loss of control and decreased agility from relying on cloud managed databases.\n\nHopefully you can see why I believe we have entered a golden age for building and running stateful applications. With the help of container attached storage like the solutions delivered by OpenEBS, you can run workloads anywhere in a completely automated way while providing a nearlt impossible level of per workload customization and performance. The net result is a boost in developer productivity and a reduction in cloud lock-in.\n\nWe are also inviting community members to write interesting blogs with hands-on content that OpenEBS and broader Kubernetes providers might find useful. So, if you have an idea for a blog post, such as doing blue/green with stateful workloads or a comparison between different Kubernetes platforms, please do get in touch. We would be happy to cross-publish on the OpenEBS blog and to otherwise collaborate with you.\n\nThanks again to the Southern California Linux community! It was an inspiring weekend.\n\nHere are the reference slides I mentioned (with a nice Monty Python GIF included as a bonus):\n\n[https://docs.google.com/presentation/d/11dNx7-HEqUg6ZeUAtaKAzfqfscmuDcXaKWbjxbM3us4/edit?usp=sharing](https://docs.google.com/presentation/d/11dNx7-HEqUg6ZeUAtaKAzfqfscmuDcXaKWbjxbM3us4/edit?usp=sharing)\n","notHasFeatureImage":false,"slug":"inspired-by-scale17x — a-short-history-of-data-storage"},{"id":58,"title":"Becoming a Kubernaut - Top Learnings in my journey of becoming a CKA","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"05-03-2019","tags":["Cka","Kubernauts","Kubernetes","MayaData","OpenEBS"],"excerpt":"I would like to tell you about my experience with the Certified Kubernetes Administrator (CKA) exam.","content":"\nHello, my fellow Kubernetes Developers,\n\nI would like to tell you about my experience with the Certified Kubernetes Administrator (CKA) exam.\n\nI was first exposed to Kubernetes when I started to work on OpenEBS, a Container Attached Storage solution.\n\nAfter working on the project for a period of time, I thought about taking the CKA certification exam. I took approximately one and a half months to practice and get my hands dirty. I completed the exam on the 28th of December 2018 and was excitedly able to pass the exam.\n\n![](https://lh6.googleusercontent.com/xjqXS1OHG4peRCNCbgIucCHlqEEgY4dzFr2vCot0pj0dRz075Xv7Ed2p9yBiqYMWLkvKOFLLIkXL-6lTtqIap981KBWebNZvfBn8SYmOstKa-SjdUruWyli92vsGatDvXt5Nveq2)\n\n## CKA is about making friends!\n\n![](https://lh6.googleusercontent.com/y-dr6deMc6gBAJs4UKCCdA_rmDTbRLyyUd5OLMj1u6_deX1JIDWVUJq-iQvBnJXKMrs3z5pqcT6FPEfaH8qJ80Gt0BdoQ5qOOEdDoy5ZI6KNfySYzr-dy52ojIb-U-W4ZiCXHJDR)\n\nWhen I was involved in developing the OpenEBS control plane, I was required to understand some basic concepts, specifically, the storage concepts of Kubernetes.\n\nI started to explore Kubernetes Docs which, in my opinion, is the greatest source for learning Kubernetes. I applied my newfound knowledge to the project. My in-house team members my company MayaData, namely Karthik, Amit, Vishnu, and Kiran just to name a few, groomed me and provided me with the right knowledge. From there, I started to develop more interest in the subject.\n\nIn mid-November 2018, I started to do an exhaustive walk through of Kubernetes docs and practiced it.\n\nObviously, there were a few things that did not make sense to me as Kubernetes covers a vast domain, e.g. Network, Linux, etc.\n\nThroughout my experience, I frequently used the Kubernetes Slack channel and asked questions. I also began to attend local meetups. From there, my journey in making friends and a Kubernetes Mastermind began. I met Suraj Narwade on one of the meetups, where he shared his experience on how he learned Kubernetes.\n\nI came to know about the Kubernaut Slack channel, space where you can join and ask about Kubernetes-related information, such as setting up CNI, performing workload operations, etc.\n\nThis journey was very exciting, and it became a journey to make new friends and, of course, this led to becoming a CKA.\n\n## About the Exam\n\n![](https://lh6.googleusercontent.com/8O1iFmqiXACiozyrrBeMHHX3GKf9HstipsrtEK7MnPyppP9kQVxxIL1aXogrCYs8fdOCYe1952aBSpqp2bokSxpMpvkn770m4wfMjuBlNEWeeETInd7cSX-l70GUq1o3QI3SonVq)\n\n- The duration of the exam is 3 hours.\n- There will be 6 Kubernetes clusters on which the questions are based.\n- There are 24 total questions.\n- You are allowed to open only Kubernetes Docs in the exam.\n\nVisit the following link to learn more about exam\n[https://www.cncf.io/certification/cka/](https://www.cncf.io/certification/cka/)\n\nRead the candidate handbook and exam tips PDF carefully. These can be found on the website link given above.\n\nTo learn more about the exam syllabus, visit the following link:\n\n[https://github.com/cncf/curriculum](https://github.com/cncf/curriculum)\n\n## Preparation and Tips\n\n![](https://lh3.googleusercontent.com/y5Y0lbm9j0gl9pMSf2pV9cAwHrLHzk2As5oReKk-gtz--IeZvCs4V1lPQeUKQImkpKQvoX3N7YGl-3OMMva5-vNzQpEYT6curv_PbRdDIZgMuKkGzx9wj4yD5CAc62xMU5YrOdid)\n\n- Go through the Kubernetes docs exhaustively, especially the concepts and task sections, so that when you see a question you can easily open the relevant topic in docs quickly.\n- Practice with Kubectl, e.g. create a deployment, service etc., without writing YAML.\n[https://kubernetes.io/docs/reference/kubectl/conventions/](https://kubernetes.io/docs/reference/kubectl/conventions/)\n\n- Avoid writing YAMLs as much as possible in the exam.\n- Don’t just read the docs, but put them into practice.\n- Know about systemctl commands.\n- Set up your own cluster from scratch using [Kubernetes Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way). I applied a little trick here; first I became familiar with basic Kubernetes constructs, played with workloads(e.g. Deployment, daemon set, etc), services etc., on an already configured Kubernetes cluster. Then I did it the hard way.\n- Some folks also advise studying books and other learning materials, but I personally did not follow any books.\n\nI will paste some useful resource links that might be helpful in addition to the Kubernetes docs.\n\n[https://suraj.pro/post/journey-to-cka/](https://suraj.pro/post/journey-to-cka/)\n\n[https://github.com/walidshaari/Kubernetes-Certified-Administrator](https://github.com/walidshaari/Kubernetes-Certified-Administrator)\n\n[https://medium.com/@sonasingh46/static-pod-in-kubernetes-e3854507655f](https://medium.com/@sonasingh46/static-pod-in-kubernetes-e3854507655f)\n\n[https://medium.com/@sonasingh46/story-of-pod-manifest-to-running-8e4b38f074ec](https://medium.com/@sonasingh46/story-of-pod-manifest-to-running-8e4b38f074ec)\n\nDefinitely join the Google Kubernaut Slack channel.\nChannel id: [https://kubernauts-slack-join.herokuapp.com/](https://kubernauts-slack-join.herokuapp.com/)\n\nAll the best for your journey in making new friends and cracking the Certified Kubernetes Administrator (CKA) exam!\n","notHasFeatureImage":true,"slug":"becoming-a-kubernaut---top-learnings-in-my-journey-of-becoming-a-cka"},{"id":59,"title":"OpenEBS releases 0.8.1 with stability fixes and improved documentation","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"25-02-2019","tags":["CAS","Cloud Native Storage","Kubernetes","OpenEBS","Stateful Applications"],"excerpt":"OpenEBS 0.8.1 is released today. In spite of being a tagged as a minor release, it has significant stability improvements and few important features. ","content":"\nOpenEBS 0.8.1 is released today. In spite of being a tagged as a minor release, it has significant stability improvements and few important features. OpenEBS community is growing and we are receiving more feedback. Most of the issues fixed in this release are in response to the feedback received from the community. In this blog article, I will capture some important highlights of the 0.8.1 release, for more details on the actual bugs fixed, see release notes.\n\n**Upgrade recommendation:**\n\nWe are recommending users of OpenEBS with cStor with Fedora 29 please upgrade to 0.8.1 as it contains an important fix that can increase the stability of the environment including avoiding possible node reboots by addressing underlying iSCSI and Kubernetes bugs. We are available to help on the community so please do get in touch if we can be of assistance.\n\n---\n\n## Highlights of 0.8.1 release:\n\n### Stability fixes:\n![image](https://cdn-images-1.medium.com/max/800/1*7BRJmxRw7IKONv7KKAgnkg.png)\n#### iSCSI discovery bug leading to Kubernetes node reboots:\n\nSome community users reported that Kubernetes cluster is becoming unstable sometimes when OpenEBS is in use. We worked with community and debugged their issues to find the root cause. After deeper analysis, we found that Kubelet continues to increase in memory usage and finally resulting in that particular node going for a reboot, causing the cluster to becoming unstable. An [issue has been reported already in Kubernetes on Kubelet](https://github.com/kubernetes/kubernetes/issues/70890), and in our case, same situation is being triggered from a different set of components, namely open-iSCSI and Fedora Kernel.\n\nHere are more details:\n\n1. We discovered a [bug Ferora 29’s Kernel](https://bugzilla.redhat.com/show_bug.cgi?id=1679565) . The kernel does not respond to a socket read request if the request size is less than the watermark (in cStor target’s case, the water mark is set to 48) if the data becomes available after the read request is made. This works without issue if the data is already available when the request arrives at the kernel.\n2. Kubelet executes an iSCSI client command for discovery of the target. During this target discovery connection, the requested size to the kernel is 16 bytes and could result in the above mentioned situation if you are on Fedora 29. If the iSCSI client or initiator does not receive the response to the discovery request, it times out after 30 seconds and the connection is retried. After 5 retries, the iSCSI client hits a bug where it returns an infinite amount of data to Kubelet, which ends up reading all of the returned data. Kubelet then consumes the entire available memory and creates an out-of-memory situation, causing the node to reboot. We have created an [open issue at open-iscsi](https://github.com/open-iscsi/open-iscsi/issues/155) for this bug as well.\n\nIn summary, when using Fedora 29 and open-iscsi, Kubelet experiences an out-of-memory issue if the iSCSI target’s TCP watermark is set to any number greater than 16. It was set to 48 by cStor’s iSCSI target. In 0.8.1, we set the watermark level to 1 and avoided the above bugs or issues (in Fedora, open-iscsi and kubelet)\n\nAll users running cStor with Fedora 29 are advised to upgrade to 0.8.1 release.\n\n**Documentation improvements**\n\nOur docs also received significant improvements. Simplified instructions and getting-started steps are provided to new users, and more troubleshooting scenarios are covered for advanced users. We have also certified cStor 0.8.1 for use with many stateful applications like GitLab, ElasticSearch, Minio, and Prometheus with the latest Helm charts and user documentation updated accordingly.\n\n![New docs — Stateful Applications with OpenEBS](/images/blog/stateful-applications-in-openebs.png)\n\nUser guides for backup and restore and RWM support are also important parts of this new documentation.\n\n![Backup and Restore Applications using OpenEBS Persistent Volumes](/images/blog/backup-and-restores.png)\n\n**Summary:**\n\nAs stated before, we recommend all cStor users upgrade to 0.8.1. We always welcome your feedback, so if you would like to see a new feature or need to report a problem, write a GitHub issue at [https://github.com/openebs/openebs/issues](https://github.com/openebs/openebs/issues)\n\nJoin our slack community at [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.be744436714e40be6d0b15e325bdf0b3.1580126064003.1580126064003.1580126064003.1&__hssc=216392137.1.1580126064004&__hsfp=3765904294)\n\nEnjoy using the 0.8.1 code! :)\n\n. . .\n\n## Other Notable Additions\n\n**Liveness Check for cStorPool Pods**\n\nSometimes when using network disks such as cloud provider disks or iSCSI disks from local SAN, it is common to experience intermittent disconnection of locally mounted disks. In such scenarios, cStorPool could move into a suspended state and require a restart of the cStorPool instance pod on that node. A liveness check probe has been added in 0.8.1 to help fix this issue. In the event of network disks undergoing a network connection flip, the disks get disconnected, cStorPool instance becomes unresponsive, liveness check probe restarts the pod and disk connections are re-established.\n\nBecause cStor volumes are replicated into three cStorPool instances, losing a cStorPool instance temporarily because of a restart causes volumes to be resilvered from the other pools.\n\n**Ephemeral Disk Support**\n\nIn 0.8.0, cStor does not provide high availability of the data if volumes are set up for replication across local disks of cloud providers such as AWS instance stores or GCP local SSDs. The local SSDs on cloud providers do not retain the data when the VMs are rebooted because a completely new VM is provisioned with newly formatted local disks. This scenario was also discussed in [another blog post](/blog/introduction-to-openebs-cstor-pools-and-considerations-during-k8s-upgrades?__hstc=216392137.be744436714e40be6d0b15e325bdf0b3.1580126064003.1580126064003.1580126064003.1&__hssc=216392137.1.1580126064004&__hsfp=3765904294#3fb6) and in the [0.8.0 docs](/docs/deprecated/releases-0x#080---dec-07-2018).\n\nEphemeral disk support is also added for cStorPools in 0.8.1. Now, users can use AWS instance stores or any other ephemeral local SSDs for setting up a highly available Kubernetes native cloud storage across zones for their stateful needs. When a Kubernetes node is lost, a new node arises with the same label and the same number of disks. OpenEBS detects this new node and a new cStorPool instance is also created. After the pool instance is set up, cStor rebuilds the volume replicas into the new cStorPool instance from the other volume replicas. During this entire process, the volume replica quorum needs to be maintained. For more details on this feature, see the docs section.\n\n**Support to Extend cStorPools onto New Nodes**\n\ncStorPools expansion feature is another good addition in this release. In 0.8.0, the pools, once set up, could not be expanded to new nodes. In 0.8.1, support is added to expand StoragePoolClaim configuration to include more nodes. For example, if you have a StoragePoolClaim config with disks from 3 nodes, and if you are want to add a new cStorPool instance, you can add the disk CRs from the new node to spc-config.yaml and mark the max pools as 4. Once you apply the new spc-config.yaml, the pool configuration will be increased to 4 cStorPool instances.\n\n**cStor targetNodeSelector Policy**\n\nSetting node selectors to targets were limited exclusively to Jiva in 0.8.0. In this release, this policy is extended to cstor as well. With this feature, you can limit the cStorController pod scheduling to one or a set of nodes in the cluster, which gives flexibility to the administrator in managing the resources across nodes.\n\n**Jiva Replicas not Loading when Data is Fragmented**\n\nWhen Jiva volumes are deeply fragmented, the number of file extends increases to a large number. In such a situation, Jiva replicas were not able to load within a specified time limit, and re-connection was also resulting in the same issue. The 0.8.1 release fixes this issue and Jiva replicas load without issue, even when they are highly fragmented.\n","notHasFeatureImage":false,"slug":"openebs-releases-0.8.1-with-stability-fixes-and-improved-documentation"},{"id":60,"title":"HA vs. DR and HA² for your DB","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"21-02-2019","tags":["Cloud Storage","DevOps","Database Administration","OpenEBS","Kubernetes"],"excerpt":"This post is intended to share observations gathered from a couple of decades worth of experience in helping enterprises build resilient systems and presented based on feedback from Kubernetes and StackStorm users over the past 4-5 years.","content":"\n*Steps to prevent cascading failures.*\n\nThis post is intended to share observations gathered from a couple of decades worth of experience in helping enterprises build resilient systems and presented based on feedback from Kubernetes and StackStorm users over the past 4-5 years. As such, mileage may vary; I myself am still learning and offer this as a way for us all to learn together, so feedback is not just welcome, it is absolutely requested.\n\nHere is what I have seen too often, and is the basis of all the principles I share below. For all of us, as we build these systems of systems with more dependencies and dynamic relationships than any one human could possible fully understand, we want to make sure that we don’t generate opaque cascading failures.\n\nIn short, Don’t Injure Yourself (DIY)\n\nFor starters, don’t make your automation so intelligent that it knows how to kill nodes that are not responding without also understanding *why* that node might be moving slowly. As I have seen before, it could be that  your load peaked the day after Thanksgiving, and by pulling the slow nodes out of the queue you are simply shortening the time before all other nodes become overwhelmed. These are the things brownouts are made of.\n\nSo, how can you avoid being thrown off the end of your own automation treadmill?\n\nHere are a few hard-learned principles that I draw upon:\n\n1. **Shift down** — tackle failure as close as possible to the failure source to limit the risk of injuring yourself with cascading failures.\n2. **Build every layer** — and  every system,  so that it is built to fail.\n3. ***Build*** *every layer* — do not think you have DR when you have HA; and don’t think you have HA just because you have one workload that spans several clusters.\n4. **Infrastructure as code –** related to #3. Always; no black boxes. The desired state is in the repo. Yes, the control loop at the center of Kubernetes will make simple decisions to trade off workloads and behaviors to the best of its ability. Grok that, and insist on everything  by deploying this approach.\n5. **Simplicity**. Yes, this is the UNIX philosophy of do one thing and do it well. No, a human cannot grok the entire system in many cases, but they must be able to grok the behavior of each subsection.\n\nLet’s look at a few FAILURE examples. Below, I have included the entire Google slide deck that contains the discussed images for you to use and review. They are even animated in this slide deck!\n\nI’ll be working off this example of a Kubernetes environment. As you can see, we have two Kubernetes clusters, each of which have many pods. I show Pod A, B and C for each, but in reality, there would be many additional pods as well. We have some stateful workloads running across those clusters.\n\n![kubernetes environment example 1](/images/blog/kubernetes-environment-1.png)\n\nNow let’s assume you have similar environments on cloud A and data center B or similar. The idea is that we have multiple locations for similar multi-cluster Kubernetes.\n\n*It may be useful for these examples to keep in mind that the fundamental architecture of a Container Attached Storage solution like OpenEBS is to run within the clusters to provide storage to stateful workloads. Essentially, this is the piece that gives each workload its own file or block access (or s3 if you run Minio on top, for example) irrespective of the underlying cloud volumes, existing storage, disks or other hardware such as NVMe. *[*https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/*](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/)\n\n![kubernetes environment example 2](/images/blog/kubernetes-environment-2.png)\n\n**Failure Condition #1: Cloud Volume or Disk Dies**\n\nStarting at the bottom of that stack, what should you do when a disk dies or a cloud volume becomes unresponsive? This is shown in by the red “X” in the image below.\n\n![Cloud Volume or Disk Dies](/images/blog/cloud-volume-or-disk-dies.png)\n\nIf you have NoSQL or some sort of scale-out SQL running, then presumably you do not need to do anything, right? I mean, you can just degrade the pod and even the cluster and it should still work, correct? Technically, that is correct, but what is the cost? What happens to your system when, for example, a ring is being rebuilt or the cache is being repopulated? How much cross-node traffic is needed?\n\nPrinciple #1 is an adaption of shift left from DevOps — in this case, it is **shift down**. You fix the dying disk or the loss of the value at that level through a simple and easy to grok RAID. The RAID should be as simple as mirroring, so that you do not waste time rebuilding or resilvering disks. In other words, don’t use Raid 5 or 6 just to save space at the cost of complexity and longer rebuild times. This could easily degrade the POD and Cluster. You should absolutely build intelligence at the bottom of the stack to ensure you survive disk failure without creating a risk of cascading failure as a result of taxing the higher levels of the stack.\n\n**Failure Condition #2: A Lost POD that is Running a Stateful Workload**\n\nThis is where HA is put to use. However, this is not the same as disaster recovery, which involves recovering an entire environment from the loss of a data center. Rather HA is intended to keep the application functioning properly with no downtime, and it should respond seamlessly to small hiccups such as pod outages.\n\nKubernetes, of course, helps here by ensuring that multiple replicas of the actual workload can be kept alive. However, Kubernetes is not necessarily proactive when it comes to keeping stateful workloads alive. This is where a properly deployed DB layer will be useful and provide a layer of HA for the application. This guarantees that reads and writes of state for the application are never interrupted. Typically, this is done by implementing a quick failover (quicker than Kubernetes might notice, without help) to a destination for writes and multiple read replicas. However, architectures amongst the dozens of DBs vary. You can read more about this subject in a recent blog I wrote on running your own DBaaS that also touches on picking the right DB from the start:\n\n[https://openebs.io/blog/running-your-own-dbaas-based-on-your-preferred-dbs-kubernetes-operators-and-containerized-storage](/blog/running-your-own-dbaas-based-on-your-preferred-dbs-kubernetes-operators-and-containerized-storage)\n\nThe TL;DR from above includes consideration about which operators you want to use for your DBs. Of course, DBs and other stateful workloads need some imperative flow to recover from failures such as Do X (log failure), Do Y (direct writes and/or reads to the active replicas), Do Z (clean up the old node), etc.\n\nHaving configured your DB in a HA manner with the help of an operator, you are now faced with a decision. But then, do you want a storage layer replica as well?\n\n![storage layer replica](/images/blog/storage-layer-replica.png)\n\nTypically, the answer we see amongst our users is *no*. However, if you have a container attached storage solution like OpenEBS, you can decide on a per-workload basis. Perhaps you have one workload in which you are concerned that the DB itself will crash, so you want to be able to immediately reschedule and redeploy the DB onto a new node with the help of Kubernetes. This way you can have the data on the node even if the DB had not been running previously on that node. Maybe you want the write ahead log to be double resilient and you want the actual data files, but you decide it is not worth the trade off, or vice-versa. I’m not sure what to call this double redundancy — maybe we can call it **HA²**.\n\nThere are definite trade-offs in this case as you would now be asking the storage to do a write and/or synchronous replication to multiple locations before acknowledging the write. While this will become less expensive when NVMe is used as a protocol and a new class of persistent memory becomes prevalent, today it can drop your write IOPS by 30–40% depending on configurations of the DB and the storage itself among other factors. Please note that this is the kind of tuning we do with the help of partners like Datastax, NuoDB, and a number of local and global systems integrators. You can always ask around in the OpenEBS community and you will almost certainly find helpers.\n\n**Failure Condition #3: What Happened to my DB?**\n\nContinuing on the prior example, if you want to be prepared for a scenario in which the DB itself disappears, then you really should have storage or some other method for keeping a copy of the state of the database. As mentioned, if you do synchronous replication or simply have the writes written in multiple places through solutions like OpenEBS, then you get every write (consistency!) but at the cost of latency and additional loading on your network. On the other hand, if your storage solution can quickly take incremental snapshots of your storage and move those snapshots somewhere else then you can commit the writes quickly, then take/move the snapshots of that data. This eliminates the latency tax, but at the cost of your recovery point objective. So, you should be prepared to lose writes with this scenario.\n\n![latency tax](/images/blog/latency-tax.png)\n\nBy the way, storage is not the only way to add a layer of resilience to your application in cases of DB failure. We often see cases in which the DB is somewhat isolated from the actual traffic by a cache layer (hello Redis!). This cache layer can take all the writes for a period of time. In this case, having a DB deployed onto a new node with all but the most recent X writes will work just fine.\n\n**Failure Condition #4: Disaster Strikes!**\n\nIn this scenario, a hurricane shows up and floods your data center. One approach is to have every single byte written in a data center or cloud asynchronously replicated to a second cloud. Unfortunately this causes suboptimal performance in Cloud A and will not guarantee that Cloud B will function properly when the load switches over.\n\n![Disaster Strikes](/images/blog/disaster-strikes.png)\n\nSo what should you do? A common pattern we have noticed involves letting the individual teams responsible for each application determine what solution best fits their project. They must have an answer if they have an SLA that leads them towards true DR. However, in the end they can decide what they want to do. It could be that they are willing to work in a degraded mode for a period of time.\n\nHere is where the ability to snapshot and move entire applications may be a good solution. We have seen OpenEBS itself being used in this way quite often with the help of Heptio’s Velero (Formerly Ark) plug-in. We are now adding capabilities to make the management of the entire workflow simpler so that each team has the necessary control to decide their approach while the broader organization can see their approaches in aggregate. This is the so-called DMaaS subproject, due for initial release in OpenEBS 0.9.\n\nAnother pattern that I heard about occurred at Netflix many years ago, and is now known as chaos engineering. At MayaData, we employ a flavor of chaos engineering in our CI/CD end to end pipelines. For instance, we publish every commit to the OpenEBS master on [OpenEBS.ci](http://openebs.ci/) and subject them to chaos engineering under a variety of stateful workloads and across a number of environments from OpenShift to managed Kubernetes services like AKS. We have open-sourced all of this, enabling tooling for stateful workloads called Litmus.\n\nThe point here is to *apply chaos engineering to your HA and DR approaches so that you can actually see them work and fail*. Everything fails occasionally , including the systems that are intended to help you survive failure. What’s important is understanding the blast radius and determining how hard or easy it is to continue operation and recover with time?\n\nWhichever method you use, keep in mind that the granularity of control that containers, microservices and Kubernetes extended via container attached storage enables is of great benefit to you. It allows responsible teams to fully grok their component in this aspect of service resilience and minimizes the blast radius ideally down to the per team scope of control: typically the microservice itself. You can think of this approach as building in fire breaks to stop cascading failures.\n\nI hope this blog is of use to those of you wrestling with ways to ensure resilience while running real (stateful) workloads on Kubernetes. The good news is that OpenEBS and other open source projects in and around Kubernetes are quickly accumulating thousands or tens of thousands of production hours and there are many experts that frequent such channels and are often ready and willing to help.\n\nSome of this experience informs our docs in the OpenEBS community, including common patterns for workloads such as Minio, MySql, Cassandra, Elastic and many others: [https://openebs.io/docs/stateful-applications/mysql](/docs/stateful-applications/mysql)\n\nAs mentioned, you can also see these and other workloads on display as each commit to master for OpenEBS is tested against them. You can even choose to inject chaos into the testing of these workloads on OpenEBS as it is developed and matured: [https://openebs.ci/workload-dashboard](https://openebs.ci/workload-dashboard)\n\nFor example, here is the Percona dashboard that shows the ability to inject failures and to generate loads. And yes, that is live. Feel free to cause chaos on our dime!\n\n![Percona dashboard 1](/images/blog/percona-dashboard-1.png)\n\nLast but not least, I’ve tried to pull this blog together into a single picture. I’ve been guilty in the past of thinking that one solution is all that is needed, whether at the storage, the cache, the automation, the DB, or the message bus layer to ensure resilience and optimize trade-offs between resilience, cost, and performance. These days I fall back on the principles I discussed above.\n\n![Percona dashboard 2](/images/blog/percona-dashboard-2.png)\n\nAs you can see , I’m not a big fan of trying to add resilience to our systems by bolting on an opaque and monolithic storage layer. This can cause unknown behavior in the midst of unexplained change, i.e. cascading failures. Rather, we recommend injecting just enough storage into your environment by extending Kubernetes itself, where appropriate, with the help of projects like OpenEBS. If you do have a pre-existing storage layer, no problem. You can make it more resilient and extend far greater granular control of storage services by running a solution like OpenEBS on top.\n\nWhatever path you take, I really do think you will be best served by taking the necessary steps to avoid the risk of cascading failures. By solving issues as close to their occurrence as possible, allowing each team to choose their own approach, planning for failure of even the backup pieces, and by participating in transparent communities, you will be able to keep moving quickly without losing data.\n\nAs always, feedback is welcomed and encouraged.Slides: [https://docs.google.com/presentation/d/1HN25JftStju9-twyqjRowuZeQU0_WNbcQH7kO_v66aU/edit?usp=sharing](https://docs.google.com/presentation/d/1HN25JftStju9-twyqjRowuZeQU0_WNbcQH7kO_v66aU/edit?usp=sharing)\n","notHasFeatureImage":false,"slug":"ha-vs.-dr-and-ha²-for-your-db"},{"id":61,"title":"Running your own DBaaS based on your preferred DBs, Kubernetes operators and containerized storage","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"14-02-2019","tags":["DevOps","Kubernetes","OpenEBS","Stackstorm","Cassandra"],"excerpt":"This blog is intended to outline some things to think about when running your own DBaaS. While I recommend considering these, I also suggest that you should think long and hard about operations automation and, of course, underlying data resilience.","content":"\n**Why would you want to do that?**\n\n![workloads onto Kubernetes](/images/blog/workloads-onto-kubernetes.png)\nNearly every day I hear about someone looking to move more workloads onto Kubernetes. This makes sense as the above data from StackOverflow indicates that Kubernetes usage has skyrocketed.\n\nThis differs from 2017, when I was mostly asked  about how running Kubernetes via OpenShift, Rancher, Kubespray, or from the source compares to buying it by the drip from the Big 3.  Now at [MayaData](http://www.mayadata.io/) and in the [OpenEBS](http://www.openebs.io/) community, we are often asked for advice about running your own Redis or Cassandra or MongoDB. The comparison these users are making is the DB as a service offering from the Big 3, or even from the DB providers themselves.\n\nSo, the first question we need to ask  is *why*? Why would you want to take on the responsibility of doing this yourself? Without digressing too far, I’ll simply summarize the answer this way: “For the same reason we run Kubernetes.” Broadly speaking, users want to run Kubernetes either to go faster or to save money — or both. They want to do so while minimizing headaches and risks. Users want everything to be Kubernetes native and to simply work.\n\nOf course, when it comes to data,  security is a top priority as well. Many enterprises such as financial services and health care providers, or those concerned about AWS as a competitor choose to run Kubernetes on their own environments. This is in part due to concerns about sensitive data being viewed by the cloud provider or by a malicious attacker. And yes, we have seen many OpenEBS users use local encryption, which seems to validate that trend.\n\n**DBaaS Alternatives**\n\nWith each of the major cloud offerings, you can buy many flavors of DBaaS. Your choice will likely depend on your technical requirements and, of course, on your relationship with these cloud vendors. Generally speaking, you will want a high-quality solution that minimizes downtime and delivers exactly what your engineers need with minimum hassle.\n\nA common way to analyze the variety of DBs is to examine the CAP theorem and what your applications need. Typically, the rule of thumb is you pick two out of three. There are many, many blogs and discussions on picking a DB, but the idea is that there is an ever-growing number of DBs, and a subset of those are run as a service by tier-one clouds.\n\n![DBaaS Alternatives](/images/blog/dbaas-alternatives.png)\n\n**A quick aside on data base flavors:**\n\nThere are many available DB solutions, many of which have little in common other than their primary task of ordering data for faster storage or access for a particular use case. There are at least eight main types:\n\n- **Key value stores** — such as the ETCD inside of Kubernetes\n- **Column stores** — such as Amazon’s Redshift. These are generally most efficient for analytics. SAP Hana is a famous proprietary in memory DB often classified as a columnar DB\n- **Document stores** — such as MongoDB\n- **Graph databases** — such as Neo4J\n- **Row stores** — Although sometimes referred to as a columnar store, Cassandra actually belongs to this category. Generally, row oriented DBs are better at writes.\n- **In memory databases** — such as Redis (please note our [VP of Product](https://redislabs.com/redisconf19/) is speaking at the April 2–3 Redis conf on considerations on Kubernetes)\n- **“Synthetic”** — such as CockroachDB, NuoDB and TiDB. These are able to retain consistency while scaling horizontally reads and writes, so arguably stretching or even making moot the ACP trade off highlighted above\n- **Time series databases** such as Influx DB, which often runs under Prometheus, arguably the most common stateful workload on Kubernetes. This is the default monitoring solution; you can read more about using OpenEBS under Prometheus in this blog post from October 2018: [https://openebs.io/blog/using-openebs-as-the-tsdb-for-prometheus](/blog/using-openebs-as-the-tsdb-for-prometheus).\n\nRelatively old and well-known SQL solutions seem to be extremely prevalent for a variety of reasons , such as the decreasing sizes of some workloads as microservices take off. Architectures increasingly use message systems like Kafka along with service meshes to pass relevant information and  data back and forth,  as opposed to dropping everything into shared NoSQL platforms. This is likely one reason that we see an enormous amount of PostgreSQL and MySQL applications running on top of OpenEBS. There are some distributed SQL systems that scale out and yet retain SQL semantics. Notable examples include TiDB and Couchase, each of which have key value stores ordering underlying data structures, and NuoDB, which is based upon a distributed memory cache. In between are solutions such as Vitess that take MySQL (or similar) and manage sharding and some of the operational automation needed to scale MySQL horizontally.\n\nYou can get a subset of these varieties from the cloud ventures as services. So, why are users increasingly running their own DBaaS? In addition to the obvious points above, more control, including running the particular DB du jour, and less spending and fewer security concerns. Whether they are well founded or not,why else might users select to build and operate their own DBaaS?\n\n**Compose a Stack that Better Serves the DB**\n\n![Compose Stack](/images/blog/compose-stack.png)\n\nDBs are themselves composed of pieces of software that have different requirements. You also have choices as to which underlying storage engines you use with each database and how you configure these storage engines as well. As an example, depending on the version and underlying storage engine being utilized, MongoDB has a journal, various data directories, and a replication log. Cassandra has commit_logs, data_files, and saved_caches. Similarly, PostgreSQL has a WAL director and data directory. With today’s container attached storage systems and the storage class construct in Kubernetes, you can easily match each component of a DB to a well-suited and tuned underlying storage component.\n\nAs an aside, if you’d like to learn about the differences between  BTrees and LSM approaches, their underlying storage engines, and related concepts tying back to the underlying computer science, you would benefit from this talk given by Damian Gryski last year:\n\n[https://youtu.be/wxcCHvQeZ-U](https://youtu.be/wxcCHvQeZ-U)\n\nSo, while the permutations of DBs are nearly endless, with many examples of many types of DBs that can be configured differently, you can use all of them on a common storage layer across any underlying physical disk or cloud volume. This can be done with the help of a solution like OpenEBS that is typically automatically customized for each permutation.\n\nYou can read more about storage for each component of a DB in the documentation of the many flavors of DBs; for example, the MongoDB documentation states:\n\n![mongoDB documentation](/images/blog/mongodb-documentation.png)\n\nIt is probably worth also noting that to deal with disk or SSD failures, the MongoDB documentation also suggests the use of RAID 10.\n\n**Easily Integrate your Disparate Data Sources**\n\nWe have seen that DBs increasingly do only one thing, and do it very well. This microservices approach to DBs relies on systems like Kafka to perform cross DB and other data source integration and streaming. This is opposed to having one external storage system provide integration of all the data with all of the attendant blast radius and difficulties that could entail. Now, much of the data may end up in Snowflake or some on premise equivalent after an ETL process, of course. Ironically, integrating all of these data sources can sometimes be easier if these DBs are known to do one thing very well within your environment.\n\n**DevOps Autonomy**\n\nOne fundamental precept of DevOps is to have small teams pick their own tools. Having each team choose their own DB solution fits this pattern, whereas having one mega DBaaS solution often does not.\n\n**What’s the catch? Operations.**\n\nHere at MayaData, we are huge believers in the power of Kubernetes as a foundation for more automated operations. However, we also see quite a bit of churn or diversity in approaches to building and running stateful workloads.\n\nCurrently, it seems like every flavor of DB and stateful workload is coming out with at least one Kubernetes operator. And while that is great, it is also somewhat concerning. Having co-founded the leader in event driven automation, [StackStorm](https://stackstorm.com/), I’ve spent many years dealing with this day two operations stuff and, well, it is hard. In my opinion, we need to embrace workflow for some of this work, whether it is Argo that is now integrated into Kubeflow or StackStorm. This sports a new container friendly and extremely powerful workflow engine called Orquesta at its core. Incidentally, [Orquesta](https://github.com/StackStorm/orquesta) can be easily embedded into other projects, and I expect and hope to see this happening more as we continue in 2019. By workflow, I mean using an engine that makes explicit the common process events such as forks, joins, conditional pauses, for example, and does so in a way that allows humans to better grok and share these patterns.\n\nAlong these lines, back in October of 2015  I helped author a blog about how Netflix and others use StackStorm for Cassandra operations.\n\n[https://stackstorm.com/2015/10/02/tutorial-of-the-week-cassandra-auto-remediation/](https://stackstorm.com/2015/10/02/tutorial-of-the-week-cassandra-auto-remediation/)\n\nIf you are interested in workflows for operations, by the way, you might find interesting this overview of the space by my StackStorm co-founder Dmitri Zimine, which also discusses the offerings from the Big 3 cloud vendors, who are positioning their workflow engines as glue to tie together functions from their serverless offerings: [https://thenewstack.io/serverless-and-workflows-the-present-and-the-future/](https://thenewstack.io/serverless-and-workflows-the-present-and-the-future/)\n\nSo — where does storage fit into all of this?\n\nWhether or not you choose to use a generalizable approach to operators  or use one of the many one-offs emerging in the Kubernetes community,  you can approach a level of generalization through the use of Container Attached Storage. While storage and related capabilities will not completely care for your stateful workloads, it can provide some common services that every database needs, and thereby allow engineers to focus on those particular aspects of each DB that need their attention.\n\nI’ve assembled common requirements into a table, as we often get questions from the community (and investors, customers, and new team members) that suggest that the lines between the dozens of DBs fighting for attention and underlying containerized storage sometimes seems lost in all the hubbub. For example, did you know that storage engines for DBs are not “really storage”? Feel free to make comments below or even better, ask questions on StackOverflow or on our Slack community: [https://slack.openebs.io](https://slack.openebs.io/).\n\n![common requirements](/images/blog/requirements.png)\n\nWe’ve recently completed the testing and creation of some recommended storage classes with one of the largest NoSQL community projects. You can always see each commit to Master of OpenEBS being tested against a variety of common data based on [OpenEBS.ci](http://openebs.ci/) as well. Grab both the storage classes and the configuration for the systems doing this testing as well if you are interested.\n\n![workload dashboard](/images/blog/workload-dashboard.png)\n\n\n**Conclusion**\n\nThis blog was intended to outline some things to think about when running your own DBaaS. While I recommend considering these, I also suggest that you should think long and hard about operations automation and, of course, underlying data resilience. At MayaData, we are actively working on a number of DBaaS implementations for enterprises and are happy to share best practices via discussions on [Slack](https://slack.openebs.io/) and elsewhere. Please do get in touch. Together, we are enabling tremendous boosts in data agility, often with the help of DBaaS deployments. Whatever your permutation of DB or other stateful workloads, we are here to do our part in helping you to achieve true data agility.\n","notHasFeatureImage":false,"slug":"running-your-own-dbaas-based-on-your-preferred-dbs,-kubernetes-operators-and-containerized-storage"},{"id":62,"title":"Introduction to OpenEBS cStor Pools and considerations during K8s upgrades","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"31-01-2019","tags":["Cloud Native Storage","Kubernetes Upgrades","Persistent Storage","Persistent Volume","Kubernetes"],"excerpt":"Developers and DevOps administrators like Kubernetes for the way it has eased the tasks in their daily lives. It has changed the way applications are designed, developed and managed. ","content":"\n> My Kubernetes was just upgraded — what happened to my data?\n\n> And what if I’m using ephemeral disks and thin provisioning to save money?\n\nDevelopers and DevOps administrators like Kubernetes for the way it has eased the tasks in their daily lives. It has changed the way applications are designed, developed and managed. However, the churn of features and capabilities of Kubernetes continues to be high and frequent upgrades to the platform are not uncommon. Kubernetes development releases are coming out on a quarterly basis and production upgrades are common every 4 to 6 months. What is more — if you are running on a Kubernetes service you typically are not 100% in control of when and how those upgrades are performed. The infrastructure software that runs on Kubernetes needs to be cognitive of these upgrades so that application developers and Kubernetes administrators do not need to do any hand-holding during such upgrades.\n\nOpenEBS is an infrastructure software component for stateful applications running on Kubernetes as it provides persistent storage volumes and associated data management capabilities. OpenEBS as cloud native or `container attached` storage has seen increased adoption since the release (last year) of the cStor storage engine, probably thanks to its support for enterprise grade snapshots and clones as well as improvements to cross availability zone awareness, etc. OpenEBS is designed in such a way that the administrators manage the storage operations in a Kubernetes native way using kubectl commands.\n\nIn this blog, I would like to drive home three points:\n\n- Introduction to cStor pools and considerations for DevOps architects\n- How cStor volumes and pools work during Kubernetes upgrades?\n- Some dos and don’ts about cStor volumes and pools during Kubernetes upgrades\n\n## Introduction to cStor Pools\n\n![cStor Pools](/images/blog/cstor-pools.png)\n\n(***cStor pools provide thin provision and save up to 50% of your storage costs on clouds.***)\n\nA cStor-Pool (Shown in the above figure as Pool1) is a set of pool instances on each participating node, where a pool instance is formed over a set of disks. The cStor target places data replicas on each of the pool instances and synchronously replicates the volume data onto those replicas. The target manages the quorum logic and rebuilds the data among replicas when they become unavailable/available. For example, if a node disappears or is rescheduled, the workload will not notice as OpenEBS will continue serving the data.\n\nEach cStor pool instance provides enterprise-grade data protection features such as data consistency, copy-on-write snapshots, RAID resiliency of disks, self-healing of data, and thin provisioning of data. The pools can be scaled within the node by adding more disks on demand or as the used capacity crosses a preset threshold such as 80%. This thin provisioning capability of cStor Pool instances typically leads to a savings of about 50% when pools are setup over cloud provider disks such as EBS, GPD, etc. and helps to limit operational overhead.\n\n## Node NOT_READY and node reboot is common in Kubernetes, and OpenEBS storage requires volume quorum.\n\nAt the beginning of this post, I talked about Kubernetes platform upgrades that are happening more frequently than one would imagine. A common phenomenon during upgrades is the reboot of the worker nodes. CPU and memory pressures are another commonly observed reason for the worker node to go into the “NOT_READY” state. In many cases, a mass upgrade of a Kubernetes environment will cause all volume replicas hosted in the cStor pool instance to go offline, which will cause the cStor volumes to operate in an online state, but in degraded mode.\n\nThis scenario when more than one or all worker nodes are offline is when “Volume replica quorum” is especially important. A volume is considered to be online (or healthy) when the replicas are in quorum. In a three-replica setup, cStor target needs two volume replicas to be available and fully synced for the volume to be in quorum. cStor marks the volume as read-only when the replicas are not in quorum. This is needed to maintain data consistency and avoid potential data loss. Simultaneous or unmanaged node reboots can cause cStor volumes to lose quorum and cause temporary data unavailability. Again, this makes sense as the worker nodes themselves are out of commission.\n\n## Node Reboot Considerations\n\nDevOps architects and Kubernetes administrators need to be watchful of this situation when Kubernetes upgrades are occurring. Instigating a planned node reboot that leads some cStor volumes to lose quorum may not be desired. This is a specific case where container attached storage like OpenEBS has a far smaller “blast radius” than typical external scale out storage. In the case of OpenEBS, you will likely lose availability to your storage for a **particular workload** when you simultaneously reboot more than one node and cause volume replicas to lose quorum. However, in the case of shared scale out external storage, such a scenario will result in data unavailability for all of your workloads at once.\n\n`Nonetheless, when performing a Kubernetes reboot the question that is often asked is \"what is the blast radius right now\"? In other words, what volumes will lose quorum if a particular node is rebooted or lost?`\n\nYou can figure this out via kubectl.\n\n### Ephemeral Disks Scenario\n\nAs hinted at above, node reboots are a distinct worry while using hosted Kubernetes services such as GKE, EKS and AKS. The default behaviour of these solutions upon reboot is that the nodes come back with completely new disks, and the total data written onto the original disks is lost. Perhaps not coincidentally, this behaviour serves as a powerful incentive to pay the cloud provider more money for their persistent storage options, even if doing so may increase your lock-in. On the other hand, if OpenEBS pools are hosted on these local disks, the reboot of a node causes the volume replicas on that pool instance to be completely rebuilt on your other volume replicas. So, your workload continues to easily access data and OpenEBS deals with the clean up behind the scenes; this pattern is fairly common on the OpenEBS community and benefits users by allowing them to use more performant and less-expensive SSDs. Please note that this auto rebuild functionality is currently supported for Jiva Pools but NOT for cStor Pools, which is one reason we have not made cStor the default storage engine for OpenEBS. The ephemeral disk/pool support for rebuilding in cStor [is coming in 0.8.1 release](/docs/concepts/cstor?__hstc=216392137.b18f31a8a021a7fe3920ac461d353400.1580126597006.1580126597006.1580126597006.1&__hssc=216392137.1.1580126597007&__hsfp=3765904294#cstor-roadmap). Until then, it is advised NOT to use cStor pools on ephemeral disks. And yes, 0.8.1 is almost available. If you have been following along in the community, for example [OpenEBS.ci](http://openebs.ci/), you have seen test results increasingly turning green.\n\n## Conclusion:\n\nThe cStor pools feature in OpenEBS is helpful in managing the storage needs of cloud native applications in a Kubernetes native way. You may want to pay attention to planned node reboots to avoid temporary data unavailability during Kubernetes upgrades. Also, DO NOT use cStor pools on ephemeral disks until OpenEBS 0.8.1 release.\n\nThanks for reading!! If you are new to OpenEBS and need help getting started, engage in our wonderful slack community at [slack.openebs.io](http://slack.openebs.io/?__hstc=216392137.b18f31a8a021a7fe3920ac461d353400.1580126597006.1580126597006.1580126597006.1&__hssc=216392137.1.1580126597007&__hsfp=3765904294).\n","notHasFeatureImage":false,"slug":"introduction-to-openebs-cstor-pools-and-considerations-during-k8s-upgrades"},{"id":63,"title":"Performance tuning, CAS and OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"30-01-2019","tags":["Cassandra","Kubectl","Kubernetes","Mayaonline","OpenEBS","Uncategorized"],"excerpt":"Now and again — ok almost every day — we get a question on the community or from our commercial users about performance.","content":"\nNow and again — ok almost every day — we get a question on the community or from our commercial users about performance.\n\n![IOPS SCREENS FROM MAYAONLINE](/images/blog/iops-screen-from-mayaonline.png)\n\nAnd our typical response is “mileage may vary.” Sometimes we also mention that there are a bunch of tunables then talk to us for help. And by the way — we love to discuss performance requirements as it shows off how CAS is different and also we get to learn much more about how OpenEBS is being used.\n\nThis blog shares some high-level points and then shares info.\n\nFirst, it is worth noting that Container Attached Storage — where the workload is married to the storage controller — is by definition **`per workload storage`**. So when you compare OpenEBS to, for example, CEPH or some other pre-Kubernetes antediluvian scale-out storage system you risk apples to banana comparisons. While CAS will be constrained by the environment itself and how many resources OpenEBS utilizes from Kubernetes — it will not be directly impacted by the addition of more and more workloads. Those workloads are more or less an island as to themselves. By comparison, a system that checks with a metadata store to determine where to write and where to read the data and stripes data across nodes, for example, will have far greater performance impacts as more workloads are added.\n\nSecondly, as I mentioned on twitter the other day, sometimes the performance you are looking to optimize should be human as opposed to workload. With OpenEBS running natively in Kubernetes you can spin up storage in seconds on a per-workload basis — and manage it in a very familiar Kubernetes way. No external distributed system run on a foundation other than Kubernetes can be as simple to run and scale. You’ll need a team of experts to run that external distributed storage system — experts that arguably should be applied to running the distributed system you are betting upon — Kubernetes itself. And you’ll inevitably find that it takes weeks to understand any external distributed state store and years to understand how it will behave in the real world. By comparison, `if you know how to run Kubernetes, you already know how to run OpenEBS.` So, if you compare the human factor of storage performance — the humans and their ability to understand and run the storage software itself — there is no real comparison between OpenEBS and any other storage. (yes, I see my fellow members of the CAS category and I know what having a kernel module means for ongoing operations :)).\n\nAlright, enough with the high-altitude view! If you are still reading, I assume you want to know the details about tuning OpenEBS.\n\nThe good news is that you can tune cStor — which is our more enterprise-focused storage engine released in 2018 — based on the characteristics of your workload. We are working on adding these configurations directly to your storage classes as part of the forthcoming 0.9 release (we are on 0.8.1 currently) to support the infrastructure as a code pattern.\n\nHere are the two main tunings that can help improve performance:\n\n1. Scheduling the application and associated cStor storage pods\n2. Tuning the cStor volume based on the application IO pattern\n\n## Scheduling\n\n![Scheduling the Application and Associated cStor Storage Pods](/images/blog/scheduling-the-application-and-associated-ctor-storage-pods.png)\n\ncStor is the OpenEBS storage engine written in C. It is Kubernetes-native and follows the same constructs as other application pods, where affinity and anti-affinity rules can be specified to localize the access to the data and to distribute the storage pods to be resilient against node and zone failures. cStor uses synchronous replication behind the scenes to keep copies of your data, so the network latency between the cStor target and its replicas will have a performance impact that needs to be accounted for when designing the system for resiliency. Note that quorum policies are configurable, so you can specify if you want cStor to acknowledge writes when received or, more typically, when received and then confirmed by one or two replicas. Perhaps not surprisingly, performance is best when the application, the cStor target, and its replicas are co-located or connected via a fast network.\n\n## Tuning the cStor Volume\n\n![Tuning the cStor Volume Based on the Application IO Pattern](/images/blog/tuning-the-cstor-volume-based-on-the-application-io-pattern.png)\n\nThe two main tunings are LU, or “lun worker,” and queue depth, which is simply the depth of the queue used in IO for the OpenEBS target. By default, when you turn up OpenEBS and use the default configurations, OpenEBS runs in a lightweight manner. However, performance improves as you add more CPU to add more workers to LU and QD.\n\nIt is possible to modify the default settings via [`kubectl exec`](https://github.com/openebs/openebs/tree/master/k8s/demo/dbench) and if you would like to try these out, feel free to contact us through the OpenEBS Slack community. With 0.9, the tunable will be available as with other storage policies that OpenEBS currently supports via storage class and persistent volume claim annotations.\n\nYou can read more about these storage policies both for cStor and for the earlier Jiva based storage engine on our docs here:\n\n[https://openebs.io/docs](/docs?__hstc=216392137.181b23812f103703b848f80cc28e7104.1575964270497.1575964270497.1575964270497.1&__hssc=216392137.1.1575964270498&__hsfp=2230078507)\n\n![Types of Storage Policies for cStor](/images/blog/types-of-storage-policies-for-cstor.png)\n(***Types of Storage Policies for cStor***)\n\nKeep in mind that optimizing performance based on environmental and workload characteristics can be fairly complex. For example, performance often depends upon whether your workload is sequential or random and of course whether it is largely read or write. Many transactional DBs are sequential write-heavy, whereas analytic workloads are often sequential read-heavy and logging is typically more random writes. Also, the block sizes matter as does the characteristics of the underlying storage device (disks vs. SSDs for example). In the case of sequential workloads on cStor, your write performance may actually **degrade** with additional workers. In that case, you would want to decrease the workers to only one thread.\n\nAgain, remember that if you are on this path we are happy to help you and learn from your experience.\nWe have a variety of automated routines coming to assist in performance testing in OpenEBS 0.9 and later releases. There is a vast amount that can be done by closing the loop between configuration and actual performance as monitored — especially when combined with an active testing or chaos engineering solution such as Litmus. While there is some performance testing in Litmus, we and others in the community are adding workload tests such as TPS or transaction tests for various DBs that are frequently run on OpenEBS (thank you, Cassandra, NuoDB, MongoDB and other friends). We are also working on additional closed-loop optimization to build into OpenEBS operators. As always, we welcome your feedback and suggestions!\n\nSo, when might you want to make changes to tune your storage for performance? Typically, we see requests for these changes in environments in which storage is the primary performance bottleneck. As an example, we have a large IoT provider that is using cStor in thousands of remote nodes. This user knows the hardware underlying those nodes and their likely workloads. They felt that it was reasonable to maximize the resource usage of OpenEBS in order to expedite local on node performance of a random heavy workload.\n\nOn the other hand, if you are most interested in saving money and are running the workloads in clouds where CPU usage is relatively expensive, then you may prefer to run OpenEBS in a lightweight manner with as few workers as possible. Keep in mind that a common use case is running OpenEBS on persistent disks from the cloud providers, converting these more performant and less-expensive ephemeral resources into the foundation for resilient storage, thereby saving money while boosting performance.\n\nAs stated before, the mileage may vary from case to case, so feel free to contact us about your unique scenario. We are seeing many ways that necessary trade-offs between performance, resource usage, and workload and storage configurations can be better optimized on a per-workload basis. If you test many permutations of storage configurations against workloads configured in many ways, you can quickly find yourself in test scenarios that could take years to complete (gulp). It is far better to decompose the problem down to workloads so that the problem becomes more tractable. For example, you can configure MySql and the underlying OpenEBS storage differently depending on the use case, and even have systems in place that alter the configuration as needed. In other words, a fundamental advantage of the CAS architecture is that it enables a level of customization and refinement that shared everything storage approaches of the past made impossible.\n\nThanks for reading. Feedback and questions are welcome.\n","notHasFeatureImage":true,"slug":"performance-tuning,-cas-and-openebs"},{"id":64,"title":"Bare metal to production stateful workloads- OpenEBS and Rancher 2.x","author":"Dave Cook","author_info":"Building Cooby Cloud at Hetzner","date":"18-12-2018","tags":["OpenEBS","Stateful Workloads","Bare metal","Docker"],"excerpt":"Besides having a really cool name, Montel Intergalactic is also a cool company to work with. I met one of their engineers in an OpenEBS Slack forum and explained to him what I was up to","content":"\nBesides having a really cool name, [Montel Intergalactic](https://www.montel.fi/index.en.html) is also a cool company to work with. I met one of their engineers in an OpenEBS Slack forum and explained to him what I was up to. The next thing was that within a week we had everything setup at Hetzner. All Montel wanted in return was this humble blog post. A note to others in the space, THIS is how you build customer loyalty and a simple gesture like this can go a long way.\n\nNeedless to say when we decide on our next vendor these guys will be at the top of the list. In fact, they’ll probably be the only ones on that list (next to [OpenEBS](https://www.openebs.io/?__hstc=216392137.052f484bfa6105a863fd21af0f05de61.1579868893254.1579868893254.1579868893254.1&__hssc=216392137.1.1579868893255&__hsfp=3765904294) of course!).\n\nThis blog will outline the detailed steps we took to achieve our goal of setting up a bare-metal Kubernetes cluster at Hetzner from scratch using Rancher 2.x, Hetzner node driver, OpenEBS, Cert-Manager, Nginx Ingress loadbalancing and the deployment of the Tutem — Hello World app to demonstrate loadbalancing and Let’s Encrypt.\n\n**NOTE:** All nodes are created with Hetzner CX21 type servers. Our root domain is `cooby.tech` and an `A` record called `rancher.cooby.tech` points to the instance IP.\n\n### Initial Setup\n\nWe’ll be creating 4 Hetzner cluster servers:\n\n1. `front-1` nginx-ingress loadbalancing (type: worker)\n2. `control-1` control plane (type: etcd, control)\n3. `worker-1` first worker (type: worker)\n4. `worker-2` second worker (type: worker)\n\n- Spin up a Hetzner CX21 instance with a Ubuntu 18.04 image\n- Generate a keypair and make sure you can ssh into the instance\n- Install docker and docker-compose from your local machine:\n\n curl https://gist.githubusercontent.com/gridworkz/d78c290c4e6fd7753dc21bb50601745a/raw/625fc3b7ddb7a654fe379f67d0bb9c6ac5b9413a/InstallDockerCompose | ssh -i ~/.ssh/ root@rancher.cooby.tech\n “/bin/bash -s”\n\n- Login to the instance and add a docker-compose.yml file to run Rancher:\n\n ![Login to instance](/images/blog/login-to-instance.png)\n\n- Run:\n\n docker-compose up -d\n\n- `Install Hetzner node-driver from:` [https://mxschmitt.github.io](https://mxschmitt.github.io/ui-driver-hetzner/)\n- `Create a node-template for frontend`\n- `Add label loadbalancer=true`\n- `Remove iscsi driver with cloud-init:`\n\n![Remove iscsi driver](/images/blog/remove-iscsi-driver.png)\n- Create a node template for workers using the same cloud-config\n\n## Application Configuration\n\nSchedule pods so that they are not running on a node with tag **loadbalancer**. This is not a hard requirement, but in our example cluster it really makes sense.\n\n![application config](/images/blog/application-config.png)\n\n## Adding Nginx-Ingress\n\nDNS scheme is configured like this:\n\n- `.cooby.tech` A <`loadbalancer machines`>\n\nWhich enables you to use:\n\n- **Example 1.** `cooby.tech` and to use nginx ingress to route it to the example1 workload. Also in the DNS server, you can do, for example, cooby.in.montel.care CNAME montel.cooby.tech and again use our nginx-loadbalancer to route cooby.in.montel.care to your desired workload.\n\n**Pro Tip:** Once an Ingress service is created it defaults to port 42. Remember to edit the services’ YAML and change it to the port your application requires. You will save yourself hours of confusion.\n\n## Let’s Encrypt Issuer\n\nThe simplest way to add Let’s Encrypt in Rancher is to install it via catalog apps.\n\nAfter that every ingress that contains the following annotations:\n\n![ingress](/images/blog/ingress.png)\n\nWill be automatically handled by Let’s Encrypt.\n\nThe first time you add an ingress you have to give the certificate a name by editing YAML and adding only the underlined parts below. Other ingresses can be added within the Rancher UI.\n\n![rancher-ui](/images/blog/rancher-ui.png)\n\n## Adding OpenEBS\n\n- Login to each node and run: *apt-get install open-iscsi*\n- Confirm that open-iscsi is active: *systemctl status open-iscsi*\n- Navigate to global -> catalog in Rancher UI and enable Helm/Stable\n- Select OpenEBS and install it via Helm Charts\n\nThis will install base OpenEBS complete with storage classes. The next step is to create a persistent volume / persistent volume claim. This can be easily accomplished by running the following YAML:\n\n![pvc](/images/blog/pvc.png)\n\nThis will create a persistent volume claim named demo-vol1-claim with 10 GB of space in the storageClass: openebs-jiva-default. This storageClassName can then be passed as a parameter to various applications that require persistent and scale-able storage.\n\n## Conclusion\n\nIn this blog, we set out to build a simple Kubernetes cluster on Hetzner server instances. The goal was to put together an introduction to building Kubernetes from scratch by using some of the great tools that are out there that make this process simpler. This should act as a good foundation for any development or POC system. In fact, we use this setup over at [Cooby Cloud](http://cooby.io/). With the goodwill gesture from [Montel Intergalactic](https://www.montel.fi/index.en.html), I was able to get this up and running within a week. Hats off to them and the great folks on the [OpenEBS](https://www.openebs.io/?__hstc=216392137.052f484bfa6105a863fd21af0f05de61.1579868893254.1579868893254.1579868893254.1&__hssc=216392137.1.1579868893255&__hsfp=3765904294) Slack channel.\n\nNext up in this series I’ll demonstrate how to install Postgres and Odoo on this design utilizing some of the wicked tools over at [XOE Solutions](https://xoe.solutions/).\n\n`Originally published at `[coobycloud.blogspot.com](https://coobycloud.blogspot.com/)` on December 19, 2018.`\n","notHasFeatureImage":true,"slug":"bare-metal-to-production-stateful-workloads--openebs-and-rancher-2.x"},{"id":65,"title":"OpenEBS Snapshots using Kubectl","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"14-12-2018","tags":["Clone","Kubernetes","Snapshot","Storage"],"excerpt":"Kubernetes has a very “pluggable” method for adding your own logic in the form of a controller using CustomResourceDefinition(CRD).","content":"\nIn Kubernetes 1.8, many of the changes involve storage. For example, volume snapshotting API has been released as a ‘prototype’ level. It is external to core Kubernetes API’s, and you can find the project under the snapshot subdirectory of the [kubernetes-incubator/external-storage](https://github.com/kubernetes-incubator/external-storage) repository. There is, however, an ongoing proposal to add them as a core Kubernetes APIs here. For a detailed explanation of the implementation of volume snapshotting, you can read the design proposal here. The prototype currently supports GCE PD, AWS EBS, OpenStack Cinder, Gluster, Kubernetes hostPath, and OpenEBS volumes. It is important to note that aside from hostPath volumes, the logic for snapshotting a volume is implemented by cloud providers and core Kubernetes storage providers. The purpose of volume snapshotting in Kubernetes is to provide a common API for negotiating with different cloud providers in order to take snapshots and restore it as new persistent volume.\n\nKubernetes has a very `pluggable` method for adding your own logic in the form of a controller using [CustomResourceDefinition](https://kubernetes.io/docs/concepts/api-extension/custom-resources/#customresourcedefinitions)(CRD). VolumeSnapshot and VolumeSnapshotData are the two new CustomResources, and will be registered with the Kubernetes API server. This [user guide](https://github.com/kubernetes-incubator/external-storage/blob/master/snapshot/doc/user-guide.md#lifecycle-of-a-volume-snapshot-and-volume-snapshot-data) provides an overview of the lifecycle of these two resources. **`Snapshot-controller`** will create a CRD for each of these two CustomResources when it starts and will also watch for VolumeSnapshot resources. It will take snapshots of the volumes based on the referred snapshot plugin. **`Snapshot-provisioner`** will be used to restore a snapshot as a new persistent volume via dynamic provisioning.\n\nThe OpenEBS operator will deploy each **`Snapshot-controller`** and **`Snapshot-provisioner`** container inside the single pod called snapshot-controller.\n\n apiVersion: v1\n kind: ServiceAccount\n metadata:\n name: snapshot-controller-runner\n namespace: default\n ---\n apiVersion: rbac.authorization.k8s.io/v1beta1\n kind: ClusterRole\n metadata:\n name: snapshot-controller-role\n namespace: default\n rules:\n - apiGroups: [\"\"]\n resources: [\"pods\"]\n verbs: [\"get\", \"list\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumes\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumeclaims\"]\n verbs: [\"get\", \"list\", \"watch\", \"update\"]\n - apiGroups: [\"storage.k8s.io\"]\n resources: [\"storageclasses\"]\n verbs: [\"get\", \"list\", \"watch\"]\n - apiGroups: [\"\"]\n resources: [\"events\"]\n verbs: [\"list\", \"watch\", \"create\", \"update\", \"patch\"]\n - apiGroups: [\"apiextensions.k8s.io\"]\n resources: [\"customresourcedefinitions\"]\n verbs: [\"create\", \"list\", \"watch\", \"delete\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshots\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshotdatas\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"services\"]\n verbs: [\"get\"]\n ---\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1beta1\n metadata:\n name: snapshot-controller\n namespace: default\n roleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: snapshot-controller-role\n subjects:\n - kind: ServiceAccount\n name: snapshot-controller-runner\n namespace: default\n ---\n kind: Deployment\n apiVersion: extensions/v1beta1\n metadata:\n name: snapshot-controller\n namespace: default\n spec:\n replicas: 1\n strategy:\n type: Recreate\n template:\n metadata:\n labels:\n app: snapshot-controller\n spec:\n serviceAccountName: snapshot-controller-runner\n containers:\n - name: snapshot-controller\n image: openebs/snapshot-controller:ci\n imagePullPolicy: Always\n - name: snapshot-provisioner\n image: openebs/snapshot-provisioner:ci\n imagePullPolicy: Always\n\nOnce **`Snapshot-controller`** is running, you will be able to see the created CustomResourceDefinitions(CRD).\n\n $ kubectl get crd\n NAME AGE\n volumesnapshotdatas.volumesnapshot.external-storage.k8s.io 1m\n volumesnapshots.volumesnapshot.external-storage.k8s.io 1m\n\n### Create Snapshot:\n\nTo create a snapshot, let’s now create the `PersistentVolumeClaim` to be snapshotted.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol1-claim\n namespace: default\n spec:\n storageClassName: openebs-standard\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nNote that this uses the openebs-standard StorageClass, which will dynamically provision an OpenEBS PersistentVolume. Let’s now create a Pod that will create data in the volume. We will take a snapshot of this data and restore it later. For example, in a busy-box application Pod, create date.txt and hostname.txt files in a mounted openEBS volume.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: busybox\n namespace: default\n spec:\n containers:\n — command:\n — sh\n — -c\n — ‘date > /mnt/store1/date.txt; hostname >> /mnt/store1/hostname.txt; tail -f /dev/null;’\n image: busybox\n imagePullPolicy: Always\n name: busybox\n volumeMounts:\n — mountPath: /mnt/store1\n name: demo-vol1\n volumes:\n — name: demo-vol1\n persistentVolumeClaim:\n claimName: demo-vol1-claim\n\nOnce the busybox pod is in a running state, we are ready to take a snapshot. After we create the VolumeSnapshot resource below, the snapshot-controller will attempt to create the snapshot by interacting with the OpenEBS snapshot APIs. If successful, the VolumeSnapshot resource is bound to a corresponding VolumeSnapshotData resource. To create the snapshot, we need to reference the PersistentVolumeClaim name in the snapshot spec that references the data we want to capture.\n\n $ cat snapshot.yaml\n apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshot\n metadata:\n name: snapshot-demo\n namespace: default\n spec:\n persistentVolumeClaimName: demo-vol1-claim\n \n $ kubectl create -f snapshot.yaml\n volumesnapshot \"snapshot-demo\" created\n $ kubectl get volumesnapshot \n NAME AGE \n snapshot-demo 18s\n\nThe conditions listed towards the bottom of the output below show that our snapshot was indeed created successfully. We can also check the snapshot controller’s logs to verify this.\n\n $ kubectl get volumesnapshot -o yaml\n apiVersion: v1\n items:\n - apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshot\n metadata:\n clusterName: \"\"\n creationTimestamp: 2018-01-24T06:58:38Z\n generation: 0\n labels:\n SnapshotMetadata-PVName: pvc-f1c1fdf2-00d2-11e8-acdc-54e1ad0c1ccc\n SnapshotMetadata-Timestamp: \"1516777187974315350\"\n name: snapshot-demo\n namespace: default\n resourceVersion: \"1345\"\n selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/fastfurious\n uid: 014ec851-00d4-11e8-acdc-54e1ad0c1ccc\n spec:\n persistentVolumeClaimName: demo-vol1-claim\n snapshotDataName: k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n status:\n conditions:\n - lastTransitionTime: 2018-01-24T06:59:48Z\n message: Snapshot created successfully\n reason: \"\"\n status: \"True\"\n type: Ready\n creationTimestamp: null\n\nWe can now look at the corresponding VolumeSnapshotData resource that was created. Notice the reference to the openebsVolume snapshot under spec, which also references the VolumeSnapshot resource we created and the PersistentVolume that from which the snapshot was taken. This PersistentVolume was dynamically provisioned by openebs-provisioner when we created our demo-vol1-claim PersistentVolumeClaim earlier.\n\n kubectl get volumesnapshotdata -o yaml\n apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshotData\n metadata:\n clusterName: \"\"\n creationTimestamp: 2018-01-24T06:59:48Z\n name: k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n namespace: \"\"\n resourceVersion: \"1344\"\n selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n uid: 2a789f5a-00d4-11e8-acdc-54e1ad0c1ccc\n spec:\n openebsVolume:\n snapshotId: pvc-f1c1fdf2-00d2-11e8-acdc 54e1ad0c1ccc_1516777187978793304\n persistentVolumeRef:\n kind: PersistentVolume\n name: pvc-f1c1fdf2-00d2-11e8-acdc-54e1ad0c1ccc\n volumeSnapshotRef:\n kind: VolumeSnapshot\n name: default/snapshot-demo\n status:\n conditions:\n - lastTransitionTime: null\n message: Snapshot created successfully\n reason: \"\"\n status: \"True\"\n type: Ready\n creationTimestamp: null\n kind: List\n metadata:\n resourceVersion: \"\"\n selfLink: \"\"\n\n### Restore Snapshot:\n\nNow that we have created a snapshot, we can restore it to a new PVC. To do this, we need to create a special StorageClass implemented by snapshot-provisioner. We will then create a PersistentVolumeClaim referencing this StorageClass to dynamically provision a new PersistentVolume. An annotation on the PersistentVolumeClaim will communicate to **`Snapshot-provisioner`** where to find the information it needs to deal with the OpenEBS API server to restore the snapshot. The StorageClass can be defined according to the code below.  Here, the provisioner field in the spec defines which provisioner should be used and what parameters should be passed to that provisioner when dynamic provisioning is invoked.\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: snapshot-promoter\n provisioner: volumesnapshot.external-storage.k8s.io/snapshot-promoter\n\nWe can now create a PersistentVolumeClaim that will use the StorageClass to dynamically provision a PersistentVolume that contains the info of our snapshot. The annotation field snapshot.alpha.kubernetes.io/snapshot refers to the VolumeSnapshot object. Snapshot-provisioner will use this resource to obtain the information it needs to perform the snapshot restore. After this, snapshot-provisioner will provision a PersistentVolume containing the contents of the snapshot-demo snapshot\n\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: demo-snap-vol-claim\n annotations:\n snapshot.alpha.kubernetes.io/snapshot: snapshot-demo\n spec:\n storageClassName: snapshot-promoter\n accessModes: ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n\nWe can check the state of demo-snap-vol-claim to see if it is Bound or not. We can also view the snapshot-provisioner logs to verify that the snapshot was restored successfully.\n\n $ kubectl logs snapshot-controller-66f7c56c4-ptjmb-c snapshot-provisioner \n ... \n ... \n I1104 11:59:10.563990 1 controller.go:813] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" for claim \"default/demo-snap-vol-claim\" created \n I1104 11:59:10.987620 1 controller.go:830] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" for claim \"default/demo-snap-vol-claim\" saved \n I1104 11:59:10.987740 1 controller.go:866] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" provisioned for claim \"default/demo-snap-vol-claim\"\n\nNow, let’s mount the `demo-snap-vol-claim` PersistentVolumeClaim onto a busybox-snapshot Pod to check whether the snapshot was restored properly. After the busybox-snapshot pod is in a running state, we can check the integrity of the files that were created before taking the snapshot.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: busybox-snapshot\n namespace: default\n spec:\n containers:\n - command:\n - sh\n - -c\n - 'tail -f /dev/null'\n image: busybox\n imagePullPolicy: Always\n name: busybox\n volumeMounts:\n - mountPath: /mnt/store2\n name: demo-snap-vol\n volumes:\n - name: demo-snap-vol\n persistentVolumeClaim:\n claimName: demo-snap-vol-claim\n\n**Clean-Up:** \n\nWe can delete the VolumeSnapshot resource, which will also delete the corresponding VolumeSnapshotData resource from K8s. This will not affect any PersistentVolumeClaims or PersistentVolumes we have already provisioned using the snapshot. Conversely, deleting any PersistentVolumeClaims or PersistentVolumes that have been used to create the snapshot or have been provisioned using a snapshot will not delete the snapshot from the OpenEBS backend. As such, we must delete them manually.\n","notHasFeatureImage":false,"slug":"openebs-snapshots-using-kubectl"},{"id":66,"title":"OpenEBS 0.8 release allows you to Snapshot and Clone cStor Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"07-12-2018","tags":["Cloud Native","Kubernetes","OpenEBS","Storage","DevOps"],"excerpt":"cStor as a new Storage Engine and Node Disk Manager (NDM) that were introduced in the previous release have proven to be quite popular with DevOps user community.","content":"\ncStor as a new Storage Engine and Node Disk Manager (NDM) that were introduced in the previous release have proven to be quite popular with DevOps user community.\n\nThe following picture shows the locations where OpenEBS is deployed. `(More on this image later in the blog.)`\n\nIMO, this rise in adoption/installations can be attributed:\n\n- Enterprises are getting serious about Agility promised by the Cloud Native Architectures, where Kubernetes is an essential piece.\n- Kubernetes mainly focuses on the workflow of attaching Storage to Applications, but not managing the Storage itself. The Agility that Cloud Native Architectures bring are somewhat lost when a dependency is created on one specialized team ( Storage IT Team) which essentially slows down the Application Delivery Process.\n- Enterprises want the same experience with their Kubernetes Clusters whether they opt for Managed Kubernetes Clusters or have their own Kubernetes Clusters either On Premise or in Public/Private Cloud.\n\n`cStor Storage Engine` — being developed in the user space doesn’t require any special Kernel taints and just works on `any platform` of user’s choice — whether On Premise or On Cloud.\n\n`NDM` — addresses another missing aspect from Kubernetes about handling various types of Disks (or Storage Devices) connected to the Kubernetes Nodes and managing them, the Kubernetes way!\n\nAs one CTO evaluating OpenEBS, put it:\n\n> `I love OpenEBS Architecture, it helps remove the layers from the application delivery process. There is no need for a dedicated storage administrator or team. I can easily scale up and down this solution.`\n\nIn the spirit of Open Source Transparency, the CTO reached out to us to share his admiration for OpenEBS and express his concern over the performance of OpenEBS running completely in Userspace. The fact that none of the other Storage Providers do this and when push comes to shove,the CTO office needs to convince the DevOps Teams to pick OpenEBS over traditional/non-cloud native Storage Systems. DevOps Teams are aware of exactly where those legacy solutions break due to their distributed nature, and they also hold a long sustained perception of Kernel being faster than Userspace.\n\nWe knew performance is going to surface as a concern sooner or later. However, we have also found that users who have evaluated OpenEBS against other solutions still pick OpenEBS for its ease of use even if they incur a slight overhead.\n\nHaving said that, performance is still a real concern when it comes to Enterprises that run heavy computational jobs, and we know it. We have been working on resolving this performance piece for quite some time now and will very soon announce the details of what we have come up with. It suffices to say that “Performance with OpenEBS will not be a concern” in the near future.\n\nBut I digress.\n\nSpeaking of 0.8 release, some interesting features that I like are:\n\n- Support for cStor Snapshot and Clones\n- Support for managing cStor Volumes via kubectl\n\nAnd there are several other enhancements and bug fixes that went in along with a huge number of PRs from external contributors to improve code readability and unit tests. A detailed release notes can be found [here](https://github.com/openebs/openebs/releases/tag/0.8).\n\nThe OpenEBS community has grokked items like: Backup/Restore of OpenEBS Volumes to S3 compatible storage using Heptio ARK, Getting around the quirks of RKE to get iSCSI working.\n\nThe E2e team has also been very busy with setting up [https://openebs.ci/](https://openebs.ci/) that is powered by OpenEBS Litmus project with Chaos injection tools etc., Sharing the details about this project warrants for another blog!\n\n—\n\nLet me stick in this blog primarily to the cStor 0.8 features:\n\n**Snapshots and Clones.** While this feature is primarily viewed from data protection and recovery aspect, it has been found to make sense for many other use cases like:\n\n- CI / CD Pipelines for Stateful Workloads. When using OpenEBS as the underlying storage, the CI tools can make use of the Snapshot functionality to freeze the state of the application in case of build failure (or success). Developers can then launch their application with the frozen state of data from the CI pipeline for further debugging or detailed analysis. Saves a ton of time from having to reproduce the issues.\n- Big Data. The Analysis Workloads typically involve downloading large amounts of data from external sources into the Kubernetes volumes (a process called — warm-up) before running the intended computation jobs. And oftentimes the data downloaded doesn’t change very frequently. With cStor volumes, the data can be downloaded once and snapshotted. The computations jobs can use the cloned volumes. Save on network bandwidth and faster processing of the data. All this without losing the granularity of isolating the different workloads from each other.\n- Improving the Onboarding experience of users. SaaS Platforms such as — exam or certification portals or development IDEs require a set of prerequisite packages to be downloaded and installed. Snapshots can help in making sure prerequisite activities are performed outside of the Onboarding workflow. When a user requests for a service, a volume can be rapidly provided by cloning from snapshots. User experience matters! 5 mins to get your IDE or exam to being are not acceptable anymore.\n\ncStor Snapshot and Clone functionality are triggered via **kubectl** itself. cStor Snapshots and Clones are reference based and are highly optimized for lowering the capacity required for maintaining large number of snapshots and clones.\n\n**cStor Volumes managed via kubectl**. cStor volumes comprise of a Target Pod that exposes iSCSI. This target pod is instrumental in making the Applications using cStor Volumes portable. The cStor Target Pod then replicates the data to cStor Storage Pools. For redundancy and high-availability, the data is replicated to cStor Pools that are located on different nodes. `(Note that multiple cStor Target Pods can write to the same set of cStor Pools).`\n![cStor Volume](/pubic/images/blog/cstor-volume.png) cStor Volume\nThe cStor Target (aka cStor Volume), cStor Volume Replica and the cStor Pools are all Kubernetes custom resources. You can use the `kubectl describe` commands to check the health or events on each of these entities. And moreover, the Disks where the cStor Pools eventually write the data to, are also represented by Kubernetes Custom Resources.\n\nPlease give it a try and looking forward to your feedback. More details usage and examples are provided in the [release notes](https://github.com/openebs/openebs/releases/tag/0.8)\n\nNow, about that world map that I shared earlier, OpenEBS release v0.8 introduced the option of sending anonymous usage analytics. The analytics collected will help answer questions like:\n\n- What is a typical lifespan of a Storage Volume in the Cloud Native Environments?\n- What is the max (or average) capacity of Volumes in Kubernetes?\n- A slightly more involved question is around the nature of the Cloud Native Applications and what it means to the storage features like replication. In other words, do applications really require a storage that does replication?. Or do they just need Local PVs?.\n\nWhat was most surprising (or not) is that users don’t always wait on releases. They also tend to pick up the master builds, if it contains what they need! The above diagram depicts the Kubernetes clusters that used the OpenEBS master build before its release.\n\nOf course, you can also turn off sending the analytics if you so desire.\n","notHasFeatureImage":false,"slug":"openebs-0.8-release-allows-you-to-snapshot-and-clone-cstor-volumes"},{"id":67,"title":"Experience with OpenEBS in this Hacktoberfest","author":"Aswath","author_info":"FullStack Dev","date":"26-11-2018","tags":["OpenEBS","Open Source","Hacktoberfest"],"excerpt":"From Hacktoberfest website, I came to know that Open Source Cafe Bangalore is conducting Hacktoberfest workshops every weekend and they are also providing some awesome goodies for our contributions. ","content":"\nFrom Hacktoberfest website, I came to know that Open Source Cafe Bangalore is conducting Hacktoberfest workshops every weekend and they are also providing some awesome goodies for our contributions. So, I decided to give it a try.\n\nI was late for the workshop on day one itself (Thanks to traffic in Bangalore!). A session about `Kubernetes` was ongoing. Since I don’t even know what a `Container` is, it was hard for me to understand what `Kubernetes` is all about. The next session was about how we can make our first PR to some open source repository. Since I already made a few PRs, I understood it.\n\n## It’s better to start with simple issues first.\n\nIn most of the repositories, people will tag the issues concerning the complexity. Thanks to the [openebs](https://medium.com/@openebs) community for tagging beginner friendly issues. So, I choose to work on a few issues tagged as `hacktoberfest` — a few of them were very simple, such as fixing the linting errors. But a few of them were a little complicated like [adding autocompletion to a CLI tool](https://github.com/openebs/openebs/issues/1987).\nTaking up the more straightforward issues initially gave me confidence. And it helped me to understand the conventions to follow for that particular repository such as signing every commit that we make.\n\n## Help is available in the community if you need.\n\nWhen I decided to work on the project OpenEBS/Maya, I struggled to set up the project on my machine. I was using MacOS, and the build instructions were written for Linux OSes. People like [Satyam Zode](https://medium.com/@satyamz) and Prateek helped me to build the project by changing some lines in the `Makefile`.\nThis project was written entirely in `Golang`. When I decided to work on this project, I didn’t even know how to write a hello world program in `Golang`. But people were there to help me whenever I got stuck.\n\n## Always consult with some maintainers before you add some external libraries to the project.\n\nI took the issue of adding bash completion to a CLI tool named mayactl, since Akshay has mentioned a library to do the same thing. I learned a bit of `Golang` from A Tour of Go. Then I tried to incorporate the bash completion library to mayactl. After a couple of failed attempts, I was able to implement the essential bash completion. In the review, [Ashutosh Kumar](https://medium.com/@sonasingh46) told me that if we add an external library to our project, we should also be able to maintain it. So, he asked me to decide on it.\n\n## You don’t need to know every bit to fix an issue.\n\nInstead of using some external library, people told me to look into the source code of kubectl to see how they implemented the bash completion. I dug into kubectl’s source and figured out the part where they performed bash completion. I copied it and made it compatible with mayactl.\n\n`Copying code is fine as long as you know what it does is.`\n\nYou don’t even need to know a language to work on an issue. You can learn the language on the go. To fix the bash completion issue, all I need to know was three things:\n\n- A little bit of `Golang`\n- Know how bash completion works\n- Where to add the code inside mayactl\n\nI didn’t know all these things before I started. I learned this by doing it.\nI was happy by knowing that this PR was selected as the best PR of the week and I won an **Amazon Echo** from OpenEBS.\n\n[Tweet](https://twitter.com/proaksh/status/1056212237361115136/photo/1?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1056212237361115136&ref_url=https%3A%2F%2Fblog.openebs.io%2Fmedia%2Fa4ac13e292477c21ec22a988cdcc3daf%3FpostId%3D64b9711a22f5)\n\nThe next issue that I took was to add some features to a Common Integration Testing Framework (CITF) that OpenEBS is planning to use for all of its projects. I thought this is difficult to work on since I don’t know how OpenEBS works. But the people like Ashutosh, [Prince Rachit](https://medium.com/@princerachit) and [Akash Srivastava](https://medium.com/@srivastavaakash) explained the essential parts of OpenEBS that are required to work on the issue. Without completely understanding it, I worked on it. And with the help from the community, I could complete the tasks.\n\n## Address the review comments properly\n\nAfter completing the task, I made a PR. People approved the changes, and they asked me to put some screenshots about the feature that I added. But I was not clear about the kind of screenshot that they needed. So, I asked them, and they clarified. I made a `GIF` about adding autocompletion in mayactl and added it along with the PR.\n\n#### `In a community, help others too`\n\nWhen we were done with issues in CITF, the next task was to test the mayactl with the CITF. I added a few test cases using CITF. Then I helped others in reviewing their code and getting it merged with the master branch. I got to see that a lot of people were facing the issues that I encountered before. So, I shared the solutions that worked for me. I got happy that I also was able to help someone who was in need.\n\nFor my contributions to OpenEBS throughout this month, I won a **laptop** from them.\n\n[Tweet](https://twitter.com/openebs/status/1057711263260717056/photo/1)\n","notHasFeatureImage":false,"slug":"experience-with-openebs-in-this-hacktoberfest"},{"id":68,"title":"Running OpenEBS On Custom Rancher Cluster","author":"Chandan Sagar Pradhan","author_info":"Software Engineer at MayaData","date":"22-10-2018","tags":["CAS","Cloud Native Storage","Kubernetes","OpenEBS","Rancher"],"excerpt":"In this blog, I will be explaining how to deploy an application in a custom Rancher cluster on an OpenEBS volume.","content":"\nIn this blog, I will be explaining how to deploy an application in a custom Rancher cluster on an OpenEBS volume. **OpenEBS** is a leading open-source storage platform that provides persistent and containerized block storage for DevOps and container environments. **Rancher** is enterprise management for Kubernetes. Before you begin, please make sure all the prerequisites are met.\n\n## Node Requirements:\n\n- Ubuntu 16.04(64-bit)\n- 4 vCPUS and 16 GB RAM\n- Docker 17.03.2\n\n## Prerequisites:\n\n- Rancher 2.0 UI installed and running.\n\nAfter the installation of the docker, I used the below command to install Rancher. For more information on installing Rancher visit Rancher docs at [https://rancher.com/docs/rancher/v2.x/en/](https://rancher.com/docs/rancher/v2.x/en/)\n\n sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable\n\n- Minimum of 1 master and 3 worker\n- (Optional) 6 nodes for the cluster ( 3 master and 3 workers)\n- *iscsiadm* should be present only on kubelet (And verification steps)\n\n### Remove iscsiadm from nodes:\n\nCheck the below commands on all worker nodes.\n\n iscsiadm -V\n docker exec kubelet iscsiadm -V\n\nSample Output:\n\n root@worker1 ~ # iscsiadm -V\n iscsiadm version 2.0–873\n root@worker1 ~ # sudo docker exec kubelet iscsiadm -V\n iscsiadm version 2.0–874\n\nIf your output is similar to the sample above, then you have to remove iscsi from the node. OpenEBS target will use the iscsi inside the kubelet. Run the commands below to remove iscsi from the node.\n\n service iscsid stop\n sudo apt remove open-iscsi\n\n### Load iscsi_tcp module:\n\nThe above step may remove the iscsi_tcp probe, and after a reboot, the node will not start the iscsi_tcp service, and OpenEBS volume mount will fail. It should be the same with the command below.\n\n lsmod | grep iscsi\n\nSampleOutput:\n\n root@worker113:~# lsmod | grep iscsi\n iscsi_tcp 20480 0\n libiscsi_tcp 24576 1 iscsi_tcp\n libiscsi 53248 2 libiscsi_tcp,iscsi_tcp\n scsi_transport_iscsi 98304 2 libiscsi,iscsi_tcp\n\nIf your output is similar to the sample above, then, you are good to go. If your output doesn’t have `iscsi_tcp`, you need to follow the below steps to load the `iscsi_tcp` module.\n\n modprobe iscsi_tcp\n\nYou can verify the same from the command below. Now the output should be similar to the sample output mentioned above\n\n lsmod | grep iscsi\n\n### Persist iscsi_tcp module to load after reboot:\n\nYou can make the kernel load iscsi_tcp automatically every time the node reboots by appending the line `iscsi_tcp in /etc/` modules.\n\nExample:\n\n # /etc/modules: kernel modules to load at boot time.\n #\n # This file contains the names of kernel modules that should be loaded\n # at boot time, one per line. Lines beginning with “#” are ignored.\n iscsi_tcp\n\nNow if all prerequisites have been met, go ahead with setting up the cluster. You can go directly to deploy the OpenEBS section if you already have a k8s cluster.\n\n### Creating a Custom k8s cluster on Rancher:\n\n- Once you have deployed Rancher, you should be able to access the UI. Login to your rancher UI using your credentials\n\n![rancher login screen](/images/blog/rancher-login-screen.png)\n\n- Now click on the global tab and then click on add cluster button. We should see the `add cluster` window.\n\nExample:\n![rancher ui add cluster](/images/blog/rancher-ui-add-cluster.png)\n\n- Under the add cluster option, click on the custom. Then give the cluster a name. You can customize the cluster under Cluster Options. You can choose Kubernetes version, Network provider and other options. Here, I have selected Kubernetes Version as V.1.11.2-rancher1–1, Network provider as Canal and Cloud Provider as none.\n\nExample:\n![rancher ui add custom](/images/blog/rancher-ui-add-custom.png)\n\n![rancher ui custom options](/images/blog/rancher-ui-custom-options.png)\n\n- Now, click on the Next button; it will open another page. On this page, you can select node roles like etcd/ Control Plane / Worker. Click on `Show advanced options.` Now add the i/p address of the node in the Internal Address section.\n\nExample:\n\n![rancher ui cluster summary](/images/blog/rancher-ui-cluster-summary.png)\n\n- Copy the command mentioned in the page to the node, which will be added to the cluster. Once you run the command, the page will show `1 new node registered.`\n\n![rancher ui more options](/images/blog/rancher-ui-more-options.png)\n\n- Now click on *Done*. Follow the same process to add more nodes to the cluster.\n- Once all the nodes are added, you can deploy OpenEBS on the cluster.\n\n### Deploy OpenEBS:\n\n**Note:** Ensure that you have met the prerequisites before installation.\n\nThe latest version of OpenEBS, i.e., 0.7 can be installed using the below steps.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.7.0.yaml\n\n### Select Your Storage Engine:\n\nYou can now choose the storage engine to provision Jiva or cStor volumes. As a cluster admin, you can provision jiva or cStor based on your requirements.\n\nHere I am going to use the Jiva storage engine.\n\n### Provisioning Jiva Storage Engine:\n\nJiva can be provisioned in your Kubernetes cluster by using the following procedure.\n\n### Verify if the OpenEBS installation is complete.\n\nOpenEBS pods are created under `openebs` namespace, default Storage Pool and default Storage Classes are created after installation.\n\nYou can get the OpenEBS pods status by running following command\n\n kubectl get pods -n openebs\n\nYou can use the default Jiva storage class in your application YAML to run the application. You can get the storage classes that are already created by using the following command.\n\n kubectl get sc\n\n### Following is an example output.\n\n NAME PROVISIONER AGE\n openebs-cstor-sparse openebs.io/provisioner-iscsi 4m\n openebs-jiva-default openebs.io/provisioner-iscsi 4m\n openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 4m\n\nOpenEBS installation will create Jiva storage pool also. It will be created by default on `/var/openebs` inside the hosted path on the nodes.\n\nYou can get the storage pool details by running the following command.\n\n kubectl get sp\n\nFollowing is an example output.\n\n NAME AGE\n cstor-sparse-pool-gjo0 5m\n cstor-sparse-pool-str9 5m\n cstor-sparse-pool-x4dm 5m\n default 5m\n\nFrom the above output, cstor-sparse-pools are default cstor engine pools whereas *default* is the jiva storage engine default pool.\n\n### Deploying applications on OpenEBS:\n\nNow I will deploy Percona DB on OpenEBS volume.\n\nYou can get the percona deployment YAML from the command below:\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-deployment.yaml\n\nYou have to edit the percona-openebs-deployment.yaml to use the jiva storage engine.\n\nUse vi command to edit the YAML file. Inside the YAML file under the PersistentVolumeClaim section, you have to update the storageClassName. You have to use the `openebs-jiva-default` storage class.\n\nExample:\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol1-claim\n spec:\n storageClassName: openebs-jiva-default \n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nRun the below command to deploy percona application.\n\n kubectl apply -f percona-openebs-deployment.yaml\n\nRun the below command to check percona pods should be running now.\n\n kubectl get pods\n\nExample Scenario 1:\n\n NAME READY STATUS RESTARTS AGE\n default-demo-vol1-claim-3213556361-ctrl-c96bdd757–4fhqq 2/2 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-glkbs 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-jxwvr 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-stcbb 1/1 Running 0 2m\n percona-86d6cf8547–7t6bz 1/1 Running 0 2m\n\nNow we are running an application successfully on an OpenEBS volume on a Rancher custom cluster.\n\n## Troubleshooting:\n\n**If application pod is stuck in containercreating for more time you have to follow the below steps**:\n\nExample Scenario 2:\n\n NAME READY STATUS RESTARTS AGE\n default-demo-vol1-claim-3213556361-ctrl-c96bdd757–4fhqq 2/2 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-glkbs 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-jxwvr 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-stcbb 1/1 Running 0 2m\n percona-86d6cf8547–7t6bz 1/1 ContainerCreating 0 2m\n\n**Cause:**\n\n- As part of the startup of the node, iscsi was already installed and running on the node.\n- Kubelet startup doesn’t start the iscsid as it is already running on the node.\n- When volume login is initiated, iscsiadm from kubelet is trying to contact the iscsid (running on the host) to initiate the connection. The version doesn’t match that results in an error from iscsid to iscsiadm and prints `12 — module not loaded`.\n\n**Recovery Step:**\n\n- Refer to the remove iscsiadm section in pre-requisites\n\n**After rebooting the nodes, the pods will stick again in container creating state.**\n\n**Cause:**\n\n- After reboot, the kernel may not load the `iscsi_tcp` module automatically.\n\n**Recovery Step:**\n\nRefer to the persist iscsi_tcp module to load after reboot section in the pre-requisites section.\n\nHopefully, this will help you to configure OpenEBS on top of Rancher 2.0. Thank you for reading, and please provide any feedback below or on twitter — [@chandan4147](https://twitter.com/chandan4147). For more details on OpenEBS installation and troubleshooting visit [https://openebs.io/docs](/docs).\n","notHasFeatureImage":false,"slug":"running-openebs-on-custom-rancher-cluster"},{"id":69,"title":"How to install IBM Cloud Private? [Updated]","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"12-10-2018","tags":["IBM","Icp","Kubernetes","OpenEBS","Persistent Storage"],"excerpt":"In this blog, I will provide step-by-step instructions on how to configure a Kubernetes-based managed private cloud using ICP.","content":"\nIt’s been some time since I wrote about [IBM Cloud Private 2.1](http://containerized.me/introduction-to-ibm-cloud-private/), although I’ve been heavily using it in my lab.\n\nImprovements from version 1.2 to 2.1 were massive. We have also noticed the changes in the OpenEBS user community, according to our surveys usage of ICP increased dramatically. The Community Edition of the ICP 3.1 came out three weeks ago. CE container images were released with a slight delay of two weeks after the enterprise version was announced. And I believe that it deserves an updated blog to talk about the steadily advancing developer experience (and my favorite new features).\n\nIn this blog, I will provide step-by-step instructions on how to configure a Kubernetes-based managed private cloud using ICP. I will also highlight the top improvements since the last version that makes me excited.\n\n![IBM Cloud Private Architecture](https://cdn-images-1.medium.com/max/800/0*9oSUd2enJ2qhcmdk.jpg)\n\n***Quick note:*** Developers who use only public cloud ask me this question frequently, “Why would you need a private cloud, and maintain it, everything on the public cloud is much easier?”. Well, you are probably not the one paying the cloud bill, right? For various tests, (some running IO intensive workloads) and learning experiments, I maintain 4 clusters (IBM Cloud Private, Rancher, Red Hat OpenShift and one installed with kubeadm) — each 3–5 nodes. That’s 16–20 nodes, exactly $1,069.05/month — I learned it all the hard way. True that education is expensive, but running the same on a few `[*cheap servers*](https://www.ebay.com/sch/i.html?_odkw=hp+proliant+g6&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313.TR1.TRC0.A0.H0.X+HP+Proliant+DL360+G6.TRS0&_nkw=+HP+Proliant+DL360+G6&_sacat=0)` adds only ~$135/month to my electric bill — soon will be almost free thanks to the `[*solar panels*](https://www.amazon.com/gp/product/B00FF1KG8U/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00FF1KG8U&linkCode=as2&tag=containerized-20&linkId=6304f51d54206e4f79da35d403ef4e96)` 😉`\n\n`When it’s not enough I also use StackPointCloud to deploy temporary clusters on AWS or GKE, etc, but I never keep them overnight unless I have to.`\n\n> [`On-premise turnkey solutions`](https://kubernetes.io/docs/setup/pick-right-solution/#on-premises-turnkey-cloud-solutions) `that allow me to create Kubernetes clusters on my internal, secure, cloud network with only a few commands like IBM Cloud Private makes my live really easy.`\n\nNow, let’s take a look at the requirements.\n\n#### **Prerequisites**\n\nMinimum requirements for a multi-node community edition cluster:\n\n**Hardware**\n\n- **Boot node:** 1x 1+ core(s) >= 2.4 GHz CPU, 4GB RAM, >=100 GB disk space\n- **Master node:** 1x 8+ cores >= 2.4 GHz CPU, 16+GB RAM, >=200 GB disk space\n- **Proxy node:** 1 or more 2+ cores >= 2.4 GHz CPU, 4GB RAM, >=150 GB disk space\n- **Worker node:** 1 or more 1+ cores >= 2.4 GHz CPU, 4GB RAM, >=150 GB disk space\n\nSince I’m using the Community Edition this time, I will be using single master and multiple workers configuration.\n\n**Software**\n\n- [Ubuntu 16.04 LTS](https://www.ubuntu.com/download/server) (18.04 LTS and RHEL 7.x is also supported)\n- Docker 18.03.1-ce\n- [IBM Cloud Private 3.1](https://hub.docker.com/r/ibmcom/icp-inception/)\n\n#### *How to install IBM Cloud Private-CE 3.1*\n\nIf you are already using the older version of the ICP, you can skip the cluster preparation steps and jump to the “Install IBM Cloud Private-CE 3.1” section after uninstalling the older version as described under “Uninstalling an older version of the IBM Cloud Private” section.\n\nWe need a few things installed before we get up and running with ICP 3.1. First, I’ll configure my Ubuntu servers and share SSH keys so that the master node can access all my other nodes. Then I’ll install Docker and after that ICP. From there, ICP will take care of my Kubernetes cluster installation.\n\n#### `Install the base O/S — Ubuntu (30–45mins)`\n\nDownload your preferred version of [Ubuntu](https://www.ubuntu.com/download). I use Ubuntu Server 16.04.3 LTS.\n\nInstall Ubuntu on all servers with default options. I used ***user/nopassword*** as username/password for simplicity.\n\nLog in to your Ubuntu host via terminal.\nEdit the `/etc/network/interfaces` file, assign a static IP and set a hostname.\n\nEdit the `etc/hosts` file, add your nodes to the list and make sure you can ping them by the hostname:\n`cat /etc/hosts`\n For my setup, this is how `hosts` file looks like:\n\n $ cat /etc/hosts\n 127.0.0.1 localhost\n # 127.0.1.1 icp3101# The following lines are desirable for IPv6 capable hosts\n #::1 localhost ip6-localhost ip6-loopback\n ff02::1 ip6-allnodes\n ff02::2 ip6-allrouters\n 10.10.0.161 icp3101\n 10.10.0.162 icp3102\n 10.10.0.163 icp3103\n 10.10.0.164 icp3104\n 10.10.0.165 icp3105\n\nPing nodes by hostname to make sure all are accessible:\n\n $fping icp21032 icp21033 icp21034 icp21035\n icp21032 is alive\n icp21033 is alive\n icp21034 is alive\n icp21035 is alive\n\nOn your Ubuntu host, install the SSH server:\n\n sudo apt-get install openssh-server\n\nNow, you should be able to access your servers using SSH. Check the status by running:\n\n $ sudo service ssh status\n ● ssh.service — OpenBSD Secure Shell server\n Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)\n Active: active (running) since Mon 2018–10–08 07:16:37 PDT; 8h ago\n Main PID: 1160 (sshd)\n Tasks: 1\n Memory: 6.8M\n CPU: 45.454s\n CGroup: /system.slice/ssh.service\n └─1160 /usr/sbin/sshd -D\n\nInstall *open-iscsi* and *curl* if it’s not already installed:\n\n sudo apt install curl open-iscsi\n\nRepeat above steps on all servers.\n\nNow, you need to share SSH keys among all nodes:\nLog in to your first node, which will be the boot node (ubuntu36), as root.\n\nGenerate an SSH key:\n\n ssh-keygen -b 4096 -t rsa -f ~/.ssh/master.id_rsa -N “”\n\nAdd the SSH key to the list of authorized keys:\n\n cat ~/.ssh/master.id_rsa.pub | sudo tee -a ~/.ssh/authorized_keys\n\nFrom the boot node, add the SSH public key to other nodes in the cluster:\n\n ssh-copy-id -i ~/.ssh/master.id_rsa.pub root;\n\nRepeat for all nodes.\nLog in to the other nodes and restart the SSH service:\n\n sudo systemctl restart sshd\n\nNow the boot node can connect through SSH to all other nodes without the password.\n\n#### `Install Docker CE (5mins)`\n\nTo get the latest supported version of Docker, install it from the official Docker repository.\nOn your Ubuntu nodes, update the apt package index:\n\n sudo apt-get update\n\nMake sure below packages are installed:\n\n sudo apt-get install \\\n apt-transport-https \\\n ca-certificates \\\n software-properties-common\n\nAdd Docker’s GPG key:\n\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n\nAdd the repository:\n\n sudo add-apt-repository \\\n “deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable”\n\nUpdate the apt package index:\n\n sudo apt-get update\n\nCheck the available versions:\n\n apt-cache madison docker-ce\n\nInstall Docker CE 18.0.3.1:\n\n sudo apt-get install docker-ce=18.03.1~ce-0~ubuntu\n\nMake sure it’s up and running after installation is complete:\n\n $ sudo systemctl status docker\n ● docker.service — Docker Application Container Engine\n Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)\n Active: active (running) since Mon 2018–10–08 07:17:42 PDT; 8h ago\n Docs: https://docs.docker.com\n Main PID: 1165 (dockerd)\n Tasks: 1442\n Memory: 7.2G\n CPU: 55min 10.946s\n CGroup: /system.slice/docker.service\n\n#### `Install IBM Cloud Private-CE 3.1 (60–75 mins)`\n\nDownload the IBM Cloud Private-CE container images:\n\n sudo docker pull ibmcom/icp-inception:3.1.0\n\nCreate an installation folder for configuration files and extract the sample config file:\n\n mkdir /opt/ibm-cloud-private-ce-3.1.0; \\\n cd /opt/ibm-cloud-private-ce-3.1.0\n sudo docker run -e LICENSE=accept \\\n -v “$(pwd)”:/data ibmcom/icp-inception:3.1.0 cp -r cluster/data\n\nAbove command creates the **cluster** directory under `/opt/ibm-cloud-private-ce-3.1.0` with the following files: `config.yaml`, `hosts`, and `ssh_key`. Before deploying ICP, these files need to be modified.\nReplace the `ssh_key` file with the private SSH key you have created earlier.\n\n sudo cp ~/.ssh/master.id_rsa /opt/cluster/ssh_key\n\nAdd the IP address of all your nodes to the *hosts* file in the `/opt/ibm-cloud-private-ce-3.1.0/cluster` directory. If you plan to run serious workloads, I recommend separating master and worker Kubernetes nodes. Since Community Edition supports single master node only, my config file looks like this:\n\n $ cat /opt/ibm-cloud-private-ce-3.1.0/cluster/hosts\n [master]\n 10.10.0.161\n [worker]\n 10.10.0.161\n 10.10.0.162\n 10.10.0.163\n 10.10.0.164\n 10.10.0.165\n [proxy]\n 10.10.0.161\n #[management]\n #4.4.4.4\n #[va]\n #5.5.5.5\n\nFinally, deploy the environment. Change directory to the cluster folder with the `config.yaml` file and deploy your ICP environment:\n\n sudo docker run — net=host -t -e LICENSE=accept \\\n -v “$(pwd)”:/installer/cluster ibmcom/icp-inception:3.1.0 install\n\n**Note:** I have tried to deploy 3.1 on my old VMs (where ICP v2.1.0.2 used to run) and my first attempt failed due to increased resource requirements compared to the previous version. If your deployment times-out while waiting for cloudant and you an error similar to below:\n\n TASK [addon : Waiting for cloudant to start] ******************************************\n *********************************************************************\n FAILED — RETRYING: TASK: addon :\n Waiting for cloudant to start (13 retries left).\n FAILED — RETRYING: TASK: addon :\n Waiting for cloudant to start\n\nDouble check your h/w resources and run the installer with verbose options to see more details:\n\n docker run -e LICENSE=accept — net=host \\\n -t -v “$(pwd)”:/installer/cluster \\\n ibmcom/icp-inception:3.1.0 install -vvv | tee -a install_log.txt\n\nI ended up increasing memory from 8GB to 16GB and disk capacity to 200GB (from 150GB) and all worked well after that.\n\nAfter a successful install you should see a message similar to the below:\n\n PLAY RECAP *********************************************************************\n 10.10.0.161 : ok=173 changed=94 unreachable=0 failed=0\n 10.10.0.162 : ok=113 changed=55 unreachable=0 failed=0\n 10.10.0.163 : ok=108 changed=51 unreachable=0 failed=0\n 10.10.0.164 : ok=108 changed=50 unreachable=0 failed=0\n 10.10.0.165 : ok=108 changed=49 unreachable=0 failed=0\n localhost : ok=265 changed=161 unreachable=0 failed=0POST DEPLOY MESSAGE ************************************************************The Dashboard URL: https://10.10.0.161:8443, default username/password is admin/admin\n Playbook run took 0 days, 0 hours, 53 minutes, 54 seconds\n\nOne thing I have noticed is that the installation completed much faster than the previous version. ICP got smarter and only pulling images that are required for the roles, also seems like taking advantage of the local image registries. My 5 node installation time dropped from 95 minutes to 53 minutes. If your deployment is successful, you should be able to access your ICP login screen by visiting `https://MASTERNODEIP:8443` (Default username/password is admin/admin).\n\n![account login](/images/blog/account-login.png)(***IBM Cloud Private Login Screen***)\n\n![IBM Cloud Private Dashboard](/images/blog/ibm-cloud-private-dashboard.png)(***IBM Cloud Private Dashboard***)\n\n![IBM Cloud Private Catalog](/images/blog/ibm-cloud-private-catalog.png)(***IBM Cloud Private Catalog***)\n\n#### `Uninstalling an older version of the IBM Cloud Private`\n\nIBM documentation is very clear with upgrade steps described [here](https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/installing/upgrade_ce.html). It is unfortunate that upgrade is only supported from the 2.1.0.3 release since I had problems installing that version. I kept my cluster at 2.1.0.2, and later clean install was the online way to go for me.\n\nI have uninstalled the existing version with the comment below:\n\n sudo docker run -e LICENSE=accept — net=host -t -v “$(pwd)”:/installer/cluster ibmcom/icp-inception:2.1.0.2 uninstall\n\nAlso removed all stopped containers:\n\n docker system prune -af\n\n#### `What’s next?`\n\nI will go over the configuration of other optional features in my next blog post as I get more familiar with the new platform.\n\n- Introduction to IBM Cloud Private #2 — What’s new in 3.1 — quick comparison vs 2.1\n- Introduction to IBM Cloud Private #3 — Catalog Applications\n- Introduction to IBM Cloud Private #4 — How to deploy workloads on OpenEBS\n- Introduction to IBM Cloud Private #5 — Monitoring IBM Cloud Private with Prometheus\n- Introduction to IBM Cloud Private #6 — Use of metrics for monitoring utilization\n- Introduction to IBM Cloud Private #7 — Contributing to the Community Charts\n- Introduction to IBM Cloud Private #8 — Chaos Engineering with Litmus\n- Introduction to IBM Cloud Private #9 — Backup your ICP cluster with OpenEBS & Heptio Ark\n\nAbove are some of the content I believe that it would be useful. If you like to see anything not covered here feel free to comment on my blog or contact me via Twitter [@muratkarslioglu](http://www.twitter.com/muratkarslioglu).\n\nTo be continued…\n\n---\n\n`Originally published at `[`Containerized Me`](http://containerized.me/how-to-install-ibm-cloud-private-updated/)`.`\n","notHasFeatureImage":false,"slug":"how-to-install-ibm-cloud-private?-[updated]"},{"id":70,"title":"Using OpenEBS for running Kubernetes stateful applications on AWS instance store disks","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"07-10-2018","tags":["Kubernetes","OpenEBS","Solutions","Stateful Applications","Tutorials"],"excerpt":"In this post, I will cover the topic of “How to set up persistent storage” using AWS instance store disks for applications running on Kubernetes clusters. ","content":"\nIn this post, I will cover the topic of “How to set up persistent storage” using AWS instance store disks for applications running on Kubernetes clusters. Instance store disks provide high performance, but they are not guaranteed to always be present in the node. This, of course, means that when the node is rescheduled you can potentially lose the data being stored. As such, there are two ways of obtaining storage on AWS for stateful applications:\n\n1. Using AWS EBS disks\n2. Using Local disks or instance store disks\n\n## Running Stateful Apps using EBS Volumes\n![Stateful Applications using EBS volumes](https://lh5.googleusercontent.com/t7qvZRLBbTPxhPJhzO6bSg6PjSD52wv2b_aDSm8G1WYai8oOc4ogMEELkN9zSmCHgY3T4XASqlguUhmLfMl0FtmCTgWD8yURvTyVZaULU9qM47L2alvFO6GQMxmd5yIenJhqHxJXRb4sHxvYNA)\n## The Problems with this Approach\n\n- When a node goes down, a new node arises as part of Auto Scaling Groups (ASG). EBS disks that are associated with the old node must be detached from the old node and attached to the new node. This process is slow and not guaranteed to work seamlessly. Also, the new EBS volume will not contain any data.\n- If the user application is capable of performing replication, then it will replicate data to this new disk. This will take more time when using large amounts of data. This will have a negative impact on performance.\n- EBS volumes are slow and users are not able to make use of faster disks such as SSDs and NVMe.\n- Slow failover means effectively no High Availability.\n- Poor I/O, unless you wish to spend a lot of unused disk space.\n\n## Why Can’t We Use Instance Disks as is for Kubernetes?\n![Stateful Applications using Instance Stores](https://lh3.googleusercontent.com/FWlk8RIHRK8tk7jvrN8jYCgV6Ho1zP8GBqiuE49y16WkaJ3gyW-8g-Z08KP0dpknnw6PNCazKMp0pCBh6AUpbJpe92Pq9Eskzb7EODaeGAYBHx4dlGBzrIX2n6iV5YMH3EFBN5hD3MycE3WeEA)\n- When a node goes down, a new node arises with its own disks, meaning any data is lost. This is because of AWS’s auto-scaling group and other policies. Per the ASG policy, the entire component associated with an instance is deleted during the termination of an EC2 instance. The user application, which has the capability to manage replication itself, has to manage the data replication across nodes.\n- What if your applications do not have this capability?\n- If a local disk fails, then your data is lost.\n\n## OpenEBS can Help Keep the Data Replicated across Nodes\n\nOpenEBS is a viable option for high availability of data, combined with the advantages of using physical disks.\n\n## How is Replication Done with OpenEBS?\n\nOpenEBS will have a minimum of 3 replicas to run the OpenEBS cluster with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk, which will come up as part of ASG. In the meantime, your workload is in accessing the live data from one of the replicas.\n![Stateful Applications using OpenEBS and Instance Stores](https://lh4.googleusercontent.com/XJCdP9q-4LX4sGr0lro6Gyj0BCMgeh85_2MTao6wnlzBAZSvtudxXplxrwDOG1zqN9n9tpltpPdHb0ssQINs2aNmQWR0M8EJRtyj30nQLMo8Zl01C-ZzTANOi0F29oUAMDvydBloOGuWme-v2w)\n## How do we Quickly Demonstrate OpenEBS on AWS?\n\n1. Setup K8s nodes to automount the disks and configure iSCSI initiators during reboot.\n2. Similarly, configure the AWS “User Data” under the launch configuration to manage the iSCSI initiators and auto mounting of disks during launch or reboot.\n3. Install OpenEBS on Kubernetes Nodes. This should be simple, and a couple of methods are discussed at the beginning of our docs, either using a Helm Chart or directly from Kubectl. More details are mentioned in [https://openebs.io/docs](/docs).\n4. Use OpenEBS Storage Classes to create Persistent Volumes for your stateful applications.\n\nBelow I provide step by step instructions that you should be able to cut, paste, and customize. As you can see, these include how to configure your AWS account for this simple POC as well.\n\n### **Summary:**\n\n> *In summary, OpenEBS can be used to easily set up stateful applications on Kubernetes where AWS instance store disks are the underlying disks. This provides good manageability, improves resilience, and allows for relatively high performance for applications.*\n\n### A detailed explanation of OpenEBS cluster deployment on AWS and rebuilding of persistent volumes are given in the section below.\n\n## Requirements\n\n- AWS account with full access to EC2, S3, and VPC\n- Ubuntu 16.04\n- KOPS tool installed\n- AWS CLI installed for AWS account access\n- SSH key generated to access EC2 instances\n\n## On Local Ubuntu Machines\n\n- Install with Ubuntu 16.04 LTS.\n- Install the kops utility package. I have followed the official[ Kubernetes documentation](https://kubernetes.io/docs/setup/custom-cloud/kops/) to complete this. I have skipped step 2 and used my VPC ID so that it can fully communicate with its components.\n- Install AWS CLI. This can be done [here](https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html).\n- Need an AWS account with full access to EC2, S3, and VPC.\n- Generate a new ssh key. If you are not familiar with this, you can check to go[ here](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/).\n\n## Installing OpenEBS Cluster in AWS\n\n**You should have access to both the AWS management console and the local Ubuntu CLI for the installation of OpenEBS.**\n\n### Perform the following operations from the AWS management console:\n\n1. Create a Virtual Private Cloud (VPC). To do this, go to VPC service and create a VPC.\n\n![Creaate a VPC](https://lh6.googleusercontent.com/Oy2n7BoGwjRClKfVuV-_ZJh_gGH1JMV8vEf32envoIktSPNLGn8vTrB4SSkhF0-OXwVgeM0Xyo5_8qiqSn88flr9EUFyOM0AntMT78pEJqSEK2yxCm3EYKf82pHO4VYttPr35-fuMgs2lHBvTw)\n2. Create an Internet gateway and associate your VPC with this Internet Gateway. This will attach internet connectivity to your VPC. All nodes under this VPC will have outside connectivity.\n![Create an Internet gateway](https://lh3.googleusercontent.com/Yd06EwIqsr8spFLw6zwShPu0Jh53jPSrWU5evOa2YVIULFvoMG5BoV3JkbgbLLPuP2ICUXGzcJs7XlxYjN9GA241bEzY5uuppT3M0KM07m4k380AP93X5NkEXlbmVTeEgRMiCOEx3oGFFMVLXA)\n\n### **Perform the following procedure from your local ubuntu machine:**\n\n1. Download the AWS CLI utility in your local machine.\n2. Connect with your AWS account by executing the following command:\n```\n aws configure\n```\n> **Note:** You must specify your AWS Access Key, Secret Access Key, Default region name, and Default output format in order to keep the configuration details.\n\n3. Create an S3 bucket to store your cluster configuration details as follows:\n```\n aws s3 mb s3://\n``` \n\n4. Export the s3 bucket details using the following command:\n```\n export KOPS_STATE_STORE=s3://\n``` \n\n5. Create the cluster using the following command:\n```\n kops create cluster — name=.k8s.local — vpc= — zones=\n``` \n\nThis will create a cluster in the mentioned zone in your provided region as part of the AWS configuration.\n\n6. Use the set of commands produced by the above step to customize your cluster configuration, such as Cluster name change, Instance group for Nodes, and master, etc. Below is an example output.\n\n**Example:**\n\nA cluster configuration has been created.\n\nSuggestions:\n\n- list clusters with: kops get cluster\n- edit this cluster with: kops edit cluster ranjith.k8s.local\n- edit your node instance group: kops edit ig — name=ranjith.k8s.local nodes\n- edit your master instance group: kops edit ig — name=ranjith.k8s.local master-us-west-2a\n\nFinally, configure your cluster with: kops update cluster name.k8s.local — yes\n\n7. Change your instance image type and the number of machines by executing the corresponding commands. The exact command needed for your cluster will be shown at the end of the previous step. The following is an example.\n\n**Example:**\n\nChange your node configuration by executing as follows:\n```\n kops edit ig — name=.k8s.local nodes\n```\nChange your master instance type and number of machines by executing as follows:\n```\n kops edit ig — name=.k8s.local master-\n``` \n\n**Note:** We used c3.xlarge as the instance type for both Master and Nodes. The number of worker nodes used is 3 and the master node as 1.\n\n8. Once the customization is done, you can update the changes as follows:\n```\n kops update cluster .k8s.local — yes\n```\n9. The above step will deploy a 3 Node OpenEBS cluster in AWS. You can check the instance creation status by finding the EC2 instance page and choosing the corresponding region.\n\n10. From the EC2 instance page, obtain each instance type Public IP.\n\n**Example:**\n![ink](https://lh5.googleusercontent.com/MR4HVU7V-kKHlBr5J1aI8GPzVzpBwX1MAsYVCyJjIVVVatEKDXQeIPm5MMaHAe3qw9fRMnZWdOti94nqfe8C39NXI1xbQ2l9Hz30UfVFDuLd5ENPp7cZ4-mg4z2haaPWlZFFUa2wncxpsXhrMw)\n11. Go to the **Launch Configuration** section on the EC2 page and take a *copy of the Launch configuration* for nodes. Select the configuration for the Node group and click on the Actions pane.\n\n**Example:**\n![link](https://lh6.googleusercontent.com/NpHS772BGXBmArOERpQyZ2Vj3U15sEq9DAhRgutrSIZxn_yTgbVgEJPaQHYX_YNtlEwxtwWPLArSzDqxrnqqhfK8IUjgIpM6vqs94nS16n7ocHgnmg9by0CAFmN3yd_IE3Zg0I7xQzR5lYdYtw)\n12. Perform changes in the **Configure Details** section as follows:\n\na. Change the new configuration name if required.\n\nb. Edit the **Advanced Details** section and add the following entry at the end of the **User data** section.\n\n #!/bin/bash\n set -x\n date\n apt-get install open-iscsi\n grep “@reboot root sleep 120;service open-iscsi restart” /etc/crontab || sudo sh -c ‘echo “@reboot root sleep 120;service open-iscsi restart” >> /etc/crontab’\n systemctl enable open-iscsi\n sh -c ‘echo “/dev/xvdd /mnt/openebs_xvdd auto defaults,nofail,comment=cloudconfig 0 2” >> /etc/fstab’\n reboot\n set -x\n umount /mnt/openebs/xvdd\n mount /dev/xvdd /mnt/openebs_xvdd\n \n\n**Example:**\n![link](https://lh4.googleusercontent.com/4qOeijgmFhAVn1Z5zWtTlWKNwZh6GyRz5a2PvEcUafuOtzmocD6AzHb_rZ_u_A265jn87iXK085k-Qyq5Nw8EelTkdKO_g9E7SVxhIY02CnVoEXDPHXSBnJTnERQQ7bAJOme5BrGlQHT19irug)\nClick **Skip** to review and proceed with the Create launch configuration.\n\n13. Go to the **Auto Scale Group** section on the EC2 page. Select the configuration for the Node group and click on the Actions pane to edit the Launch Configuration. Change the existing one with the new Launch Configuration and save the new setup.\n\n**Example:**\n![link](https://lh5.googleusercontent.com/590yBjH_Yp97YgfGpYh4eGM-mnj2qTQr6XcjQVgzW7lzGo5-0P7fismEoBP6_XNw-2xxD4yFYPiEiqTZQk1KEEJl5TgMo9JlWEEE6tKzngV2T0H3nTu-_cZ0iXNY-UInoG6ytvlheSr8qm1eSA)\n14. SSH to each node using its public key as follows:\n```\n ssh -i ~/.ssh/id_rsa admin@\n```\n15. SSH to all Nodes where OpenEBS will be installed, and perform the following commands to install the iSCSI packages and auto mounting of the local disk during reboot.\n```\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n sudo cat /etc/iscsi/initiatorname.iscsi\n sudo service open-iscsi status\n sudo sudo sh -c ‘echo “/dev/xvdd /mnt/openebs_xvdd auto defaults,nofail,comment=cloudconfig 0 2” >> /etc/fstab’\n grep “@reboot root sleep 120;service open-iscsi restart” /etc/crontab || sudo sh -c ‘echo “@reboot root sleep 120;service open-iscsi restart” >> /etc/crontab’\n sudo reboot\n ``` \n\n16. SSH to Master Node and perform the following commands to clone the OpenEBS YAML file and deploy it.\n```\n wget\n https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml\n \n wget\n https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml\n``` \n\n17. Edit `openebs-operator.yaml` and add the following entry. This will create a storage pool on one of the local disks attached to the hosts. Refer to OpenEBS Storage Pools for more information.\n```\n — -\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: jivaawspool\n type: hostdir\n spec:\n path: “/mnt/openebs_xvdd”\n — -\n``` \n\n18. Edit `openebs-storageclasses.yaml` by adding the following entry in your corresponding storage class:\n```\n openebs.io/storage-pool: “jivaawspool”\n```\n**Example:**\n\n — -\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-percona\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/storage-pool: “default”\n openebs.io/jiva-replica-count: “3”\n openebs.io/volume-monitor: “true”\n openebs.io/capacity: 5G\n openebs.io/storage-pool: “jivaawspool”— -\n — -\n\n19. Apply `openebs-operator.yaml` by executing the following command:\n```\n kubectl apply -f openebs-operator.yaml\n```\n20. Apply openebs-storageclasses.yaml by executing the following command:\n```\n kubectl apply -f openebs-storageclasses.yaml\n```\n21. Deploy your application YAML that will be created on the local disk.\n\n**Example:**\n```\n kubectl apply -f percona-openebs-deployment.yaml\n```\n22. To check the status of applications and Jiva Pods, use the following command:\n```\n kubectl get pods -o wide\n``` \n\nAn output similar to the following should be displayed.\n\n NAME READY STATUS RESTARTS AGE IP NODE\n \n percona-7f6bff67f6-cz47d 1/1 Running 0 1m 100.96.3.7 ip-172–20–40–26.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-ctrl-84bcf764d6–269rj 2/2 Running 0 1m 100.96.1.4 ip-172–20–62–11.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-bzjq4 1/1 Running 0 1m 100.96.1.5 ip-172–20–62–11.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-lpz2k 1/1 Running 0 1m 100.96.2.8 ip-172–20–32–255.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-rqnr7 1/1 Running 0 1m 100.96.3.6 ip-172–20–40–26.us-west-2.compute.internal\n \n\n23. Obtain the status of PVC using the following command:\n```\n kubectl get pvc\n```\nAn output similar to the following is displayed.\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-vol1-claim Bound pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6 5G RWO openebs-percona 3m\n \n\n24. View the status of PV using the following command:\n```\n kubectl get pv\n```\nThe output from the above command will be similar to the following.\n\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6 5G RWO Delete Bound default/demo-vol1-claim openebs-percona 3m\n \n\n*Now you have deployed OpenEBS with AWS local disk in your k8s environment. You will see the advantages of both low latency local disk use and fault-tolerant architecture ensured by OpenEBS.*\n\nHopefully, this helps you to better understand the benefits of using OpenEBS on top of AWS. Thank you for reading and please provide any feedback below or via Twitter — @ranjithr005\n","notHasFeatureImage":false,"slug":"using-openebs-for-running-kubernetes-stateful-applications-on-aws-instance-store-disks"},{"id":71,"title":"Contributing to mayactl cli in Hacktoberfest 2018","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"05-10-2018","tags":["OpenEBS","Minikube","Hacktoberfest","Ubuntu","Kubernetes"],"excerpt":"Hacktoberfest is an excellent platform for first-time, open-source hackers to start their journey. mayactl is a tool that allows for contribution at any level.","content":"\nHacktoberfest is an excellent platform for first-time, open-source hackers to start their journey. mayactl is a tool that allows for contribution at any level. Users can find easy fixes or find features with medium complexity that take anywhere from one to three weeks for those who are new to OpenEBS.\n\nmayactl is a command line tool designed for configuring/debugging OpenEBS. It helps with the status of various resources related to OpenEBS on a given Kubernetes cluster. For those who want to contribute to these type of features, this blog post will help you get started with mayactl. We will cover:\n\n- How to set up the development environment of OpenEBS\n- Tips on hacking mayactl\n- How to test the changes before you send a PR\n\nPre-requisites:\n\n- Beginner knowledge of Kubernetes\n- Basic knowledge of GO Language\n- Basic understanding of how OpenEBS functions. You can get started with OpenEBS [docs](/docs?__hstc=216392137.a9b75e72cb4b227999b631a7d9fb75d2.1579850476359.1579850476359.1579850476359.1&__hssc=216392137.1.1579850476359&__hsfp=3765904294) and the OpenEBS white paper\n\n## Setting up the Development Environment for mayactl\n\nSo, let’s start with the setup for the development environment of mayactl.\n\nSome pre-requisites for this:\n\n1. Any Linux system (Ubuntu recommended). For installing Ubuntu 18.04, you can follow this great article [here](https://linuxconfig.org/how-to-install-ubuntu-18-04-bionic-beaver).\n2. Golang. A very good article regarding this is available [here](https://linuxconfig.org/install-go-on-ubuntu-18-04-bionic-beaver-linux).\n3. Git. As with Ubuntu, the git tool comes preinstalled, but if it is missing, you can install it by sudo apt install git.\n4. Docker. Docker can be easily installed on Ubuntu by following the commands given below:\n\n sudo apt-get update\n sudo apt-get install apt-transport-https ca-certificates curl software-properties-common\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository \\\n \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable\"\n sudo apt-get update\n sudo apt-get install docker.io\n\n5. Open-iscsi. For installing open-iscsi on Ubuntu, use the below commands.\n\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n\n6. Kubectl. For installing kubectl on Ubuntu, you can visit this [link](https://kubernetes.io/docs/tasks/tools/install-kubectl/).\n\n7. Minikube. Install the latest version of minikube from [here](https://github.com/kubernetes/minikube/releases).\n\n## Cloning and making maya resources ready \n\nOnce you have completed the previous steps, it’s time to [fork](https://guides.github.com/activities/forking/) and [clone](https://help.github.com/articles/cloning-a-repository/) the [openebs/maya](https://github.com/openebs/maya) repo in your system. The repo should be cloned in following path `$GOPATH/src/github.com/openebs/`.\n\n mkdir -p $GOPATH/src/github.com/src/openebs\n cd $GOPATH/src/github.com/src/openebs\n git clone https://github.com//maya.git\n\nThese commands will clone maya into your local system.\n\nOnce successful, go into the maya project directory using cd maya, and run the make command in the terminal. This will build all the images of the openebs component and other important binaries, along with mayactl.\n\nOnce the command is complete, we will be ready with the mayactl binaries and other maya for testing. You can verify this by running docker images.\n\nIf the output is similar to the code above, then you are good to go.\n\n## Minikube Setup for Running a Single-Node Cluster\n\nWhat is Minikube?\n\nMinikube is a tool that enables users to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your local system (Learn [more](https://kubernetes.io/docs/setup/minikube/)).\n\nTo spin up a kubernetes cluster with minikube, you can use the following command:\n\n sudo minikube start --vm-driver none\n\nYou can verify your cluster by using the kubectl command that was previously installed. Running kubectl get nodes should generate an output similar to the one given below.\n\n NAME STATUS ROLES AGE VERSIONminikube Ready master 26m v1.10.0\n\n## Setting up OpenEBS on Minikube\n\nOnce the Kubernetes cluster is ready, it’s time to install OpenEBS. Installing OpenEBS on Kubernetes is very easy; just apply the openebs-operator to the cluster using the following command:\n\n kubectl appy -fhttps://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml.\n\nApplying the openebs-operator creates OpenEBS resources in Kubernetes. We can obtain the openebs-related resources in the OpenEBS namespace, and we can verify it by typing the following command:\n\n ashishranjan738@Ashish-PC:~$ kubectl get pods -n openebs\n \n NAME READY STATUS RESTARTS AGE\n \n cstor-sparse-pool-ni3o-8485bf44cd-fndj6 2/2 Running 0 5m\n \n maya-apiserver-674c77bb4c-dhnw5 1/1 Running 0 6m\n \n openebs-ndm-rd7hf 1/1 Running 0 6m\n \n openebs-provisioner-6d6c9ccf75-s57j8 1/1 Running 2 6m\n \n openebs-snapshot-operator-5cdd4ddc46-c2qj5 2/2 Running 0 6m\n\n## Obtaining Access to mayactl\n\nOnce all of the pods are in the Running state, and once the maya-apiserver pod is in the Running state, we can access the mayactl binary by accessing the maya-apiserver pod’s shell. We can gain access to mayactl by using the following command:\n\n kubectl exec -it -n openebs\n\n(e.g the pod name in my case is `maya-apiserver-674c77bb4c-dhnw5`). This will open the pod’s terminal over your system terminal. Typing `mayactl version` should produce something similar to this:\n\n bash-4.3# mayactl version\n \n Version: 0.7.0-unreleased\n \n Git commit: 0fa41c79dde0f6378056093d62494cd2fbb1eea4\n \n GO Version: go1.11\n \n GO ARCH: amd64\n \n GO OS: linux\n \n m-apiserver url: http://172.17.0.5:5656\n \n m-apiserver status: running\n \n Provider: KUBERNETES\n \n You can explore the current features of mayactl using the mayactl -help.command. Now we will look at where you can find the mayactl source code for hacking and how you can test the mayactl binary.\n\n## Exploring mayactl Code\n\nThe source code of mayactl can be found in cmd/mayactl in the maya [repository](https://github.com/openebs/maya/tree/master/cmd/mayactl). mayactl uses [cobra](https://github.com/spf13/cobra) cli framework for its implementation, so knowledge of this will be very helpful when starting with [good-first issues](https://github.com/openebs/openebs/labels/good%20first%20issue).\n\n## Building the mayactl Binary\n\nAfter playing with the code, it’s time to compile the code to binary and test it against the maya-apiserver. For building the mayactl binary, you can use the make mayactl command. This will trigger the build process, and the newly-built binary can found in `bin/maya` under the maya directory that you have cloned. This will generate an output of ls -ltr `bin/maya/` -ltr :\n\n## Copying the mayactl Binary to a maya-apiserver Pod\n\nOnce we are ready with the binary, we can run a test by copying it to the maya-apiserver pod’s `/tmp` folder. For copying the binary, run the following command:\n\n kubectl cp bin/maya/mayactl :/tmp -n openebs.\n\nThis will copy your mayactl binary to the maya-apiserver pod.\n\n## Testing the mayactl Binary\n\nIn order to test the mayactl binary, again get into the pod's shell using\nkubectl exec -it -n openebs. Then access the tmp directory using cd `/tmp`. You can run your new mayactl binary by prefixing the `./` before mayactl using `./mayactl` version.\n\n## A Bash Shortcut for the Above Process\n\nHmm…, so that much work is required just to test my small code change?\n\nDon’t worry, here is a shortcut that you can use to build and copy the mayactl binary to a maya-apiserver pod:\n\n make mayactl && kubectl cp bin/maya/mayactl `kubectl get pods -n openebs | grep maya-apiserver | awk {'print $1'}`:/tmp -n openebs && kubectl exec -it `kubectl get pods -n openebs | grep maya-apiserver | awk {'print $1'}` bash -n openebs\n\nRunning the above command will build the mayactl binary and copy to the maya-apiserver pod under `/tmp` directory.\n\n## Raising a PR\n\nOnce finished with the hacking, you can raise the PR to the maya repo. Before raising, however, it’s a good idea to have a look at the [contributer’s guideline](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md). Also, don’t forget to [sign](https://github.com/probot/dco/blob/master/README.md) your commit.\n\n## Joining the Slack Channel\n\nYou can reach us for any queries/announcements on our [hacktoberfest2018](https://openebs-community.slack.com/?redir=%2Fmessages%2Fhacktoberfest2018%2F) Slack channel.\n\n## Hacking begins…\n\nSo, what are you waiting for? Start your hacking now by picking some issues from [here](https://github.com/openebs/openebs/issues)! Contribution to the *hack-fest* can get you a chance to get into the *OpenEBS contributor's list* and a chance to win some exciting goodies.\n\n## Helpful References\n\n[Weekly Contributor’s Meet videos](https://www.youtube.com/watch?v=y-7mwbdVgwk&list=PLMvwgr-vV2NVdgQsU6sfFZXPMuiQGi4Hh)\n","notHasFeatureImage":false,"slug":"contributing-to-mayactl-cli-in-hacktoberfest-2018"},{"id":72,"title":"Setting up persistent volumes in RWX mode using OpenEBS","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"02-10-2018","tags":["OpenEBS","Solutions","wordpress","Nfs","Cloud Native Storage"],"excerpt":"Many stateful applications like WordPress require persistent storage in Read-Write-Many or RWX mode. OpenEBS is popular in the open-source community for its ease of use, and its simplistic design for pluggable storage engines.","content":"\nMany stateful applications like WordPress require persistent storage in Read-Write-Many or RWX mode. OpenEBS is popular in the open-source community for its ease of use, and its simplistic design for pluggable storage engines.\n\nCurrently, block volume or iSCSI support is provided natively by OpenEBS. Here, native iSCSI support means that the iSCSI stack is part of OpenEBS project and has full support for synchronous replication, granular monitoring, day 2 storage operations like backup restore, etc. iSCSI is typically used in Read-Write-Once or RWO mode which is common for all block storage engines. There have been multiple inquiries from the OpenEBS community about RWX support from OpenEBS, in other words, support for NFS. Though native support for NFS is being considered, it is already possible today to provision RWX mode storage using OpenEBS and to use it for applications like WordPress that require RXW mode.\n\n![openebs-nfs](/images/blog/wordpress-with-openebs-over-nfs.png)\n\nIn this blog, an example of WordPress is taken to show how OpenEBS storage volumes are exposed in RWX mode through the use of NFS in between WordPress and the Jiva volumes of OpenEBS.\n\nWordPress, when deployed on Kubernetes, requires both shared storage volumes on NFS and block storage volumes on iSCSI. The shared storage is required to store the core WordPress content or the admin managed content so that all the WordPress PODs can share the same data quickly. When PODs are spawned by Kubernetes on the fly to service more traffic, the PODs need to initialize quickly and require the core data to be available close to the application and in RWX mode. As an extremely high percentage of the shared storage traffic is Read traffic, the RWX volume need not be highly performant for writes and therefore the data can be served through NFS sitting in front of an iSCSI volume.\n\nThe typical deployment of a scalable WordPress application is shown in the below diagram.\n\n![wordpress-deployment-architecture](/images/blog/wordpress-deployment-architecture.png)\n\nAs shown in the above, OpenEBS can be used to serve the storage volumes in both RWO and RWX modes. The NFS storage volume for WordPress is served through the Kubernetes external storage plugin “[nfs](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs)”. The block storage for the database needs of WordPress is provided through OpenEBS JIVA volumes. If distributed databases like Percona or MariaDB are used then a common approach is that the database is deployed as a StatefulSet for horizontal scalability and Jiva volumes are deployed as a single replica. Alternatively one can use MySQL with jiva persistent volume replicating to three copies for enhanced resiliency.\n\n## Configuration details of PVC's and Storage Classes\n\n![pvc-and-storage-classes](/images/blog/pvc-and-storage-classes.png)\n\nThe PVC construct openebs-nfs-pvc and storage class construct openebs-nfs-sc are used to create an NFS share in RWX mode to be consumed by the WordPress pod. The deployment spec of nfs-provisioner uses an OpenEBS PVC claim which dynamically provisions the JIVA volumes in RWO mode and mounts them inside the nfs-provisioner pod. The entire process can take under 10–15 seconds.\n\n## A note on the required size of the NFS volume and the provisioned size of Jiva volume\n\nAs we are providing RWX volume over and above the RWO Jiva volume, iSCSI space overhead needs to be considered. It is recommended that the OpenEBS storage class specifies 10% more space than what is required by the NFS PVC.\n\nFor example, if WordPress NFS PVC spec specifies 100G as the required storage of the NFS volume, it is recommended to request 110G as the required storage from the OpenEBS Jiva volume.\n\n### TL;DR\n\nShared storage can be provisioned in RWX mode on OpenEBS through a single PVC request.\n\n## YAML spec examples:\n\nThe same YAML spec can be found in the Kubernetes external storage NFS plugin. The YAML specs below show how to provision the OpenEBS iSCSI storage in the nfs-provisioner pod automatically.\n\n## YAML spec for NFS provisioner security policy\n\nGitHub Code Snippet:\n\n \n kind: PodSecurityPolicy\n metadata:\n name: openebs-nfs-provisioner\n spec:\n fsGroup:\n rule: RunAsAny\n allowedCapabilities:\n - DAC_READ_SEARCH\n - SYS_RESOURCE\n runAsUser:\n rule: RunAsAny\n seLinux:\n rule: RunAsAny\n supplementalGroups:\n rule: RunAsAny\n volumes:\n - configMap\n - downwardAPI\n - emptyDir\n - persistentVolumeClaim\n - secret\n - hostPath\n\nBy applying the above YAML we will create PodSecurityPolicy for the NFS server.\n\n## YAML spec for ClusterRoleBinding for NFS provisioner\n\nGitHub Code Snippet:\n\n kind: ClusterRole\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-nfs-provisioner-runner\n rules:\n - apiGroups: [\"\"]\n resources: [\"persistentvolumes\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumeclaims\"]\n verbs: [\"get\", \"list\", \"watch\", \"update\"]\n - apiGroups: [\"storage.k8s.io\"]\n resources: [\"storageclasses\"]\n verbs: [\"get\", \"list\", \"watch\"]\n - apiGroups: [\"\"]\n resources: [\"events\"]\n verbs: [\"create\", \"update\", \"patch\"]\n - apiGroups: [\"\"]\n resources: [\"services\", \"endpoints\"]\n verbs: [\"get\"]\n - apiGroups: [\"extensions\"]\n resources: [\"podsecuritypolicies\"]\n resourceNames: [\"nfs-provisioner\"]\n verbs: [\"use\"]\n ---\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-run-nfs-provisioner\n subjects:\n - kind: ServiceAccount\n name: openebs-nfs-provisioner\n # replace with namespace where provisioner is deployed\n namespace: default\n roleRef:\n kind: ClusterRole\n name: openebs-nfs-provisioner-runner\n apiGroup: rbac.authorization.k8s.io\n ---\n kind: Role\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-leader-locking-nfs-provisioner\n rules:\n - apiGroups: [\"\"]\n resources: [\"endpoints\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\"]\n ---\n kind: RoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-leader-locking-nfs-provisioner\n subjects:\n - kind: ServiceAccount\n name: openebs-nfs-provisioner\n # replace with namespace where provisioner is deployed\n roleRef:\n kind: Role\n name: openebs-leader-locking-nfs-provisioner\n apiGroup: rbac.authorization.k8s.io\n\nThe above YAML will successfully create role bindings for the NFS provisioner. So it’s time to configure NFS provisioner YAML to use OpenEBS volumes and apply them to the Kubernetes cluster.\n\n## YAML spec for NFS provisioner deployment\n\nGitHub Code Snippet:\n\n ---\n apiVersion: v1\n kind: ServiceAccount # Creating a service account for openebs-nfs-provisioner\n metadata:\n name: openebs-nfs-provisioner\n ---\n apiVersion: v1\n kind: Service # Creating a service for openebs-nfs-provisioner\n metadata:\n name: openebs-nfs-provisioner\n labels:\n app: openebs-nfs-provisioner\n spec:\n ports:\n - name: nfs\n port: 2049\n - name: mountd\n port: 20048\n - name: rpcbind\n port: 111\n - name: rpcbind-udp\n port: 111\n protocol: UDP\n selector:\n app: openebs-nfs-provisioner\n ---\n apiVersion: apps/v1\n kind: Deployment # Creating deployment for openebs-nfs-provisoner\n metadata:\n name: openebs-nfs-provisioner\n spec:\n selector:\n matchLabels:\n app: openebs-nfs-provisioner\n replicas: 1\n strategy:\n type: Recreate\n template:\n metadata:\n labels:\n app: openebs-nfs-provisioner\n spec:\n serviceAccount: openebs-nfs-provisioner\n containers:\n - name: openebs-nfs-provisioner\n image: quay.io/kubernetes_incubator/nfs-provisioner:latest\n ports:\n - name: nfs\n containerPort: 2049\n - name: mountd\n containerPort: 20048\n - name: rpcbind\n containerPort: 111\n - name: rpcbind-udp\n containerPort: 111\n protocol: UDP\n securityContext:\n capabilities:\n add:\n - DAC_READ_SEARCH\n - SYS_RESOURCE\n args:\n - \"-provisioner=openebs.io/nfs\" # Name of the provisioner\n env:\n - name: POD_IP\n valueFrom:\n fieldRef:\n fieldPath: status.podIP\n - name: SERVICE_NAME\n value: openebs-nfs-provisioner\n - name: POD_NAMESPACE\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n imagePullPolicy: \"IfNotPresent\"\n volumeMounts:\n - name: export-volume\n mountPath: /export\n volumes:\n - name: export-volume\n persistentVolumeClaim:\n claimName: openebspvc\n ---\n apiVersion: v1\n kind: PersistentVolumeClaim # Creating PVC for openebs-nfs-provisoner to mount on it\n metadata:\n name: openebspvc\n spec:\n storageClassName: openebs-jiva-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: \"110G\"\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass # Creating storage class for applications to point to openebs-nfs-provisioner\n metadata:\n name: openebs-nfs\n provisioner: openebs.io/nfs \n parameters:\n mountOptions: \"vers=4.1\" # TODO: reconcile with StorageClass.mountOptions\n\nIn the above YAML under the args users can see we are providing -provisioner=openebs.io/nfs this means that the NFS provisioner will claim volumes for those applications whose PVC is pointing this provisioner and these applications will be deployed on top of NFS provisioner, I will explain it further. Now at the end of the YAML, you can see that we are creating a PVC under which the storage class is pointing to openebs-jiva-default this means we want to use Jiva storage engine for provisioning volume for NFS provisioner.\n\n## YAML Spec for accessing NFS from WordPress\n\nGitHub Code Snippet:\n\n - -\n apiVersion: v1\n kind: Service\n metadata:\n name: wordpress-mysql\n labels:\n app: wordpress\n ....\n # Specs for the application\n ....\n volumes:\n  - name: wordpress-persistent-storage\n persistentVolumeClaim:\n claimName: openebs-nfs\n  - -\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: openebs-nfs\n annotations:\n volume.beta.kubernetes.io/storage-class: \"openebs-nfs\" # Pointing to OpensEBS-NFS-Provisioner\n spec:\n accessModes:\n   - ReadWriteMany # Making this RWX to mount on multiple applications\n resources:\n requests:\n storage: 100G\n\n## Summary\n\nOpenEBS storage can be used in RWX mode by deploying NFS provisioner in front of OpenEBS storage volume which is in RWO mode. This model is scalable for applications like WordPress. If you know or have any applications or workloads that require RWX access mode in the PVC, please comment below. Thank you for reading and connect with us at [OpenEBS Community](https://slack.openebs.io/?__hstc=216392137.d133cc61899b42c9a03d23aff802a1df.1579851978877.1579851978877.1579851978877.1&__hssc=216392137.1.1579851978878&__hsfp=3765904294)or at our Twitter handle [@openebs](http://twitter.com/openebs).\n","notHasFeatureImage":false,"slug":"setting-up-persistent-volumes-in-rwx-mode-using-openebs"},{"id":73,"title":"Using OpenEBS as the TSDB for Prometheus","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"02-10-2018","tags":["Cloud Native Storage","Kubernetes","OpenEBS","Prometheus","Solutions"],"excerpt":"Prometheus has become one of the favorite tools for monitoring metrics of applications and infrastructure in the cloud native space, especially when using Kubernetes; ","content":"\nPrometheus has become one of the favorite tools for monitoring metrics of applications and infrastructure in the cloud native space, especially when using Kubernetes; After all, Prometheus was the second project adopted by the CNCF after Kubernetes itself and in August of 2018 was the second project to graduate from CNCF as well.\n\nGiven this lineage, setting up Prometheus on Kubernetes is quite sstraightforward. You can read much more about getting going on Prometheus as well as how it compares to alternatives such as Graphite or InfluxDB on their documentation: [https://prometheus.io/docs/introduction/overview/](https://prometheus.io/docs/introduction/overview/)\n\nOne of the challenges with Prometheus is how to setup and manage the storage for it. The default behavior of Prometheus is to simply have each node store data locally however this of course exposes the user to the loss of data stored locally when the local node goes down.\n\n## Storage for Prometheus : Local and Remote\n\nPrometheus 2.4 documentation categorizes the storage for Prometheus into two types. Local storage and Remote storage. When using local storage, data monitoring data is written in TSDB format and when using Remote storage, the data is written through storage adaptors and the format in which the data is stored is not controlled by Prometheus. Here is a quick infographic about Local and Remote storage usage for Prometheus.\n![Prometheus storage — Local vs Remote](https://cdn-images-1.medium.com/max/800/0*N9cDfVd6xzRsAFK3)\nPrometheus does a great job of simplifying the data format to be written to local TSDB. However, there are some disadvantages when choosing local raw disks as the final place to store these TSDB entries, including:\n\n1. No replicated data.\n2. The data is limited to a single node. If the Prometheus server crashes or needs to be restarted for any reason, the service has to wait till it comes up on the same node.\n3. Not scalable in terms of adding more capacity or performance dynamically.\n\nBy using OpenEBS volumes as the local storage for Prometheus on Kubernetes clusters, each of the above drawbacks is addressed.\n\nBy way of introduction, OpenEBS is the easiest to use and most widely adopted open source container attached storage for Kubernetes. OpenEBS provides per application storage controller, and the storage is scalable per application. With these features and the synchronous replication capability, OpenEBS is well suited for using it as local storage for Prometheus. You can read much more about OpenEBS on [https://openebs.io/docs](/docs?__hstc=216392137.66231d5c5a305b0a901ac5f8b69a7f4c.1580127122062.1580127122062.1580127122062.1&__hssc=216392137.1.1580127122063&__hsfp=3765904294). It should take very little time to deploy and begin to use OpenEBS once you have a Kubernetes environment. Also, it may be worth noting that at MayaData in addition to sponsoring OpenEBS we also use Prometheus extensively and also WeaveCortex for our own support.\n\nThe following (mentioned [here](https://prometheus.io/docs/prometheus/latest/storage/)) is no longer true when using OpenEBS as local storage\n\n“If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. However, you can also try removing individual block directories to resolve the problem. This means losing a time window of around two hours’ worth of data per block directory. Again, Prometheus’s local storage is not meant for durable long-term storage.”\n\n## Use OpenEBS volume as Prometheus TSDB for durability and scalability\n\n![Replicated TSDB storage for Prometheus](/images/blog/replicated-tsdb-storage-for-prometheus.png)\n\nOpenEBS volumes are replicated synchronously, in that way, data is protected and is always made available against either a node outage or a disk outage. Replica and quorum policies are independently controlled so that, for example, OpenEBS can be configured to acknowledge a write when N replicas respond, where N is <= replica count. OpenEBS can configure Kubernetes in such a way that a Prometheus pod is always scheduled in one of the three nodes that the OpenEBS replica resides. To make the data even more highly available, the OpenEBS replicas can reside in different Availability Zones of Kubernetes which in general (if in the cloud) are mapped to physical AZs of the cloud provider.\n\n## Scalability of storage\n\nThe fact that capacity cannot be added easily as needed is one of the top limitations of using local storage for Prometheus. OpenEBS helps to address this issue; capacity can be added on-demand and on-the-fly to OpenEBS volumes. So, your Prometheus local storage never runs out of storage space with OpenEBS.\n\n![On the fly storage expansion with OpenEBS](/images/blog/scalability-of-storage.png)\n\nLocal disks are grouped into storage pools, and OpenEBS volumes are carved out from these storage pools. Because of the pooling concept, you can start with small size for the storage volume and expand dynamically on demand without any service disruption i.e., on-the-fly. Disk failures are protected against an optional RAID configuration among the disks. Even an entire pool failure (can happen when a disk fails on a stripe group for example or when storage infrastructure fails) is non-fatal in OpenEBS as the rebuilding of the data happens from other replicas. So you have both local redundancy and cross-host redundancy by using OpenEBS.\n\n## What about performance? WAL support\n\nBecause OpenEBS is a pluggable, containerized architecture it can easily use different storage engines that write data to disk or underlying cloud volumes; the two primary storage engines are Jiva and cStor. Write cache support is available in the cStor storage engine. With WAL support, the write performance of Prometheus increases significantly.\n\n## How to configure OpenEBS as storage for Prometheus?\n\nJust build the storage class and update your Prometheus YAML. Your Prometheus is up and running with highly available TSDB storage with OpenEBS volumes automatically created and configured.\n\nPay attention below to the storage class parameters in the storage class. **`openebs-prometheus-sc`**\n\n ---\n # prometheus-deployment\n apiVersion: extensions/v1beta1\n kind: Deployment\n metadata:\n name: openebs-prometheus\n namespace: openebs\n spec:\n replicas: 1\n .\n .\n .\n volumeMounts:\n # prometheus config file stored in the given mountpath\n - name: prometheus-server-volume\n mountPath: /etc/prometheus/conf\n # metrics collected by prometheus will be stored at the given mountpath.\n - name: cstor-prometheus-storage-volume\n mountPath: /prometheus\n volumes:\n # Prometheus Config file will be stored in this volume\n - name: prometheus-server-volume\n configMap:\n name: openebs-prometheus-config\n # All the time series stored in this volume in form of .db file.\n - name: cstor-prometheus-storage-volume\n persistentVolumeClaim:\n claimName: cstor-prometheus-storage-volume-claim\n ---\n #PersistentVolumeClaim for prometheus\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: cstor-prometheus-storage-volume-claim\n namespace: openebs\n spec:\n storageClassName: openebs-prometheus-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 300G\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-prometheus-sc\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n\n #(Optional) Below 3 lines required only if to schedule the target pods deployed on the labeled nodes\n - name: TargetNodeSelector\n value: |-\n node: appnode\n provisioner: openebs.io/provisioner-iscsi\n \n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: prometheus-pool\n spec:\n name: prometheus-pool\n type: disk\n maxPools: 3\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get disks`\n disks:\n diskList:\n # For Eg: the below disk is from Node1 AZ1\n - disk-184d99015253054c48c4aa3f17d137b1\n - disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # For Eg: the below disk is from Node2 AZ2\n - disk-806d3e77dd2e38f188fdaf9c46020bdc\n - disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # For Eg: the below disk is from Node3 AZ3\n - disk-bad1863742ce905e67978d082a721d61\n - disk-d172a48ad8b0fb536b9984609b7ee653\n ---\n\n## Summary:\n\nUsing OpenEBS as storage for Prometheus on Kubernetes clusters is an easy and viable solution for production-grade deployments. Try using OpenEBS and see if it lives up to the expectation of being the easiest-to-use cloud-native storage project in Kubernetes ecosystem. Join our [slack](http://slack.openebs.io/?__hstc=216392137.66231d5c5a305b0a901ac5f8b69a7f4c.1580127122062.1580127122062.1580127122062.1&__hssc=216392137.1.1580127122063&__hsfp=3765904294) channel if you need help or to share your success story.\n","notHasFeatureImage":false,"slug":"using-openebs-as-the-tsdb-for-prometheus"},{"id":74,"title":"Storage Scheduling goes mainstream in Kubernetes 1.12","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"01-10-2018","tags":["Kubernetes","Open Source","OpenEBS","Storage Containers"],"excerpt":"With every new release of Kubernetes, I find myself in awe and also at ease with the choices we made early on to marry OpenEBS with Kubernetes.","content":"\nWith every new release of Kubernetes, I find myself in awe and also at ease with the choices we made early on to marry OpenEBS with Kubernetes.\n\nThere are a lot of Storage management capabilities being built into Kubernetes such as PV/PVC metrics, PV resize, PV Quota, Pod Priority Classes, and the Mount Propagation features that greatly enhance OpenEBS. However, I am especially excited about a couple of features that came in with Kubernetes 1.12:\n\n- Taint Nodes based on Conditions\n- Topology Aware Storage Provisioning\n\n## Taint Nodes based on Conditions ([#382](https://github.com/kubernetes/features/issues/382)):\n\nOpenEBS Volume services comprise of a Target Pod and a set of Replicas. When a node that is running the Target pod is unable to serve the Pods — the Target Pod needs to be evicted and rescheduled immediately. If you are using OpenEBS 0.6 or higher the Target Pods have the following eviction tolerations specified.\n\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 0\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 0\n\nUp until now, the above tolerations take effect only when the Kubernetes TaintNodeByCondition feature was enabled via alpha gate. With K8s 1.12, this feature has moved to beta and is enabled by default. Along with this feature, the performance improvements done in scheduling will help in faster rescheduling of the OpenEBS Target pod and so help to keep the data storage by OpenEBS highly available.\n\n## Topology Aware Dynamic Provisioning ( [#561](https://github.com/kubernetes/features/issues/561))\n\nThis feature mainly benefits the Persistent Volumes that have connectivity or access limitations such as Local PVs that cannot be accessed by Pods outside of the node or Cloud PVs like EBS and GPD that cannot be accessed outside of the zone in which they were provisioned. OpenEBS never had this limitation so this connectivity or access benefit is not really needed by the OpenEBS community.\n\nHowever, I am excited about some of the new capabilities that are now added to the StorageClass and PVC that can benefit OpenEBS volumes as well.\n\nFor instance, OpenEBS storage classes also can be set with *volumeBindingMode* of *WaitForFirstConsumer* as follows:\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-standard\n provisioner: openebs.io/iscsi\n volumeBindingMode: WaitForFirstConsumer\n\nThe PVCs provisioned with the above StorageClass will contain the information of the Node selected by the scheduler to launch the associated Pod in the following PVC annotation.\n\n## `volume.kubernetes.io/selected-node`\n\nOpenEBS can then use the above annotation to determine the preferred node where the Target Pod can be scheduled. This provides a simpler way to schedule the Target Pods on the same Node as the Application Pod.\n\nThis feature decidedly is an important step towards making Storage PVs a first-class citizen in scheduling. This feature helps with the initial provisioning of the volumes — I am excited about the enhancements that are planned in this area, such as the ability for the Volume Plugins to specify the preferred location where the Application Pods can be scheduled.\n\nWhile this release made progress in making Storage a first-class citizen of Kubernetes schedulers, a lot of work is underway to make the Storage Lifecycle easy to manage with the upcoming support in CSI of Snapshot, Clone, Backup and Recovery.\n\nIt feels great to be associated with Kubernetes and OpenEBS and the incredible team that is helps the DevOps teams sleep better.\n\n—\n\nBtw, it is [Hacktoberfest](https://hacktoberfest.digitalocean.com/) This is a great time to become part of the Open Source community. OpenEBS is also participating in this year’s [Hacktoberfest](/blog/celebrate-hacktoberfest-2018-with-openebs-?__hstc=216392137.073930d2db558f65dd6e9df2ff66b40e.1580119414166.1580119414166.1580119414166.1&__hssc=216392137.1.1580119414166&__hsfp=3765904294) with a friendly team that is available to help you get started with your contributions to OpenEBS and other projects.\n\nAs always, feel free to reach out to us on Slack or add comments below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.073930d2db558f65dd6e9df2ff66b40e.1580119414166.1580119414166.1580119414166.1&__hssc=216392137.1.1580119414166&__hsfp=3765904294).\n","notHasFeatureImage":false,"slug":"storage-scheduling-goes-mainstream-in-kubernetes-1.12"},{"id":75,"title":"Celebrate Hacktoberfest 2018 with OpenEBS !!","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"27-09-2018","tags":["Digital Ocean","Hacktoberfest 2018","Hacktoberfest","OpenEBS","Open Source"],"excerpt":"The 5th edition of Hacktoberfest is around the corner and it will be the second Hacktoberfest experience for the OpenEBS community. All of us are excited!","content":"\nThe 5th edition of [Hacktoberfest](https://hacktoberfest.digitalocean.com/) is around the corner and it will be the second [Hacktoberfest](https://openebs.io/hackfests/hacktoberfest-2018) experience for the OpenEBS community. All of us are excited!\n\nIn August 2017, OpenEBS community began growing and it was a beginning to building a strong foundation for an open source project. We were first introduced to Hacktoberfest by friends & peers at the DigitalOcean Bangalore Meetup and were interested in participating in it. We enlisted OpenEBS as one of the projects participating in Hacktoberfest 2017. We were pleasantly surprised by the participation and enthusiasm that Hacktoberfest attracts from developers around the world. The result was that the OpenEBS community grew by 20+ new contributor friends with ~100 PRs.\n\nWe are excited to announce that OpenEBS would be participating in Hacktoberfest 2018 as well! Look for latest updates [here](https://openebs.io/hacktoberfest)\n\n## How do you get started with Hacktoberfest?\n\nGitHub makes it really simple for contributors to get started. Just search for the tag “[hacktoberfest](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)” at the GitHub level. Within a few minutes, you can filter tens of issues to start working on that which matches your interest. First timers can look out for “good first time” tag. Anyways, here are the quick steps for you if you are interested in participating.\n\n1. Register at [DigitalOcean](https://hacktoberfest.digitalocean.com/) website and provide your mailing address so that their system knows you are participating and can send you the t-shirt and stickers later. Well, we all want them, a matter of pride 🙂\n2. Next, filter the issues with [hacktoberfest](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93) tag, and further filter out based on language, documentation, good first issue, etc\n3. Pick up few issues and let the author of the issues know that you want to fix them and start the interaction.\n4. Send a PR and with the continued interaction, you will eventually be able to get your PR merged. After sometime, your t-shirt and stickers will arrive. In the meantime enjoy working on more issues you enjoy solving.\n\n## Get additional swag from OpenEBS\n![OpenEBS swag for Hactoberfest 2018](https://cdn-images-1.medium.com/max/600/1*BXesj2ROGdmUH3vXHkRaRA.png)\nWhen your PR to any OpenEBS repositories gets merged, we will get in touch with you to fill up a form to send a special edition t-shirt designed for Hacktoberfest.\n\nWe also send some of our popular mule stickers along with your t-shirt. The best PR also gets to win a new laptop. Additional details can be found [here](https://openebs.io/hackfests/hacktoberfest-2018?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294).\n\n## Onsite workshops in Bangalore\n![Open Source Cafe at HSR Layout Bangalore](/images/blog/openebs-cafe.png)\n\nWe hope to have even greater success in this year’s edition of Hacktoberfest. With that intention, we are hosting onsite workshops on all Saturdays in October in our Open Source Cafe — Bangalore.\n\n**Schedule on the day of workshops**\n\n***10:00–10:30 AM — GitHub 101***\n\n***10:30–11:00 AM — Hacktoberfest 101***\n\n***11:00–3:00 PM — Hacktoberfest Onsite Hackathon***\n\n***11:00–12:00 PM — On-demand 30 minute sessions such as Kubernetes 101 / OpenEBS 101***\n![Onsite Hacktoberfest workshops in Bangalore](/images/blog/event-details.png)\n\nTo help beginners start their open-source journey, there will be 101-level courses in the mornings. All workshop participants will receive help from our team in getting started, and they all get to win the special edition t-shirts. Bring your own hacking gear, such as laptops!\n\n*As you will be lost in code while you are at the open-source cafe, we will bring the lunch to your hacking table :) Submit a PR on those days at the venue, collect the hacktoberfest t-shirt from the OpenEBS team, and if you order a beverage, that beer is on us. Yes, bring it on! :)*\n\n### **MULE HAS FRIENDS**\n\nThis year our theme for Hacktoberfest is “**MULE HAS FRIENDS**”. OpenEBS collaborates contributions from cloud native storage and data management. We want to drive the hacktoberfest enthusiasm to additional projects such as Weave Scope, Rancher LongHorn and Heptio ARK. This October, we are looking to help and receive more contributions for these open-source projects as well.\n\nHappy hacking!!\n\n### **Quick links**\n\n[Hacktoberfest 2018 - DigitalOcean](https://hacktoberfest.digitalocean.com/)\n\n[Hacktoberfest is a month-long celebration of open source software.](https://hacktoberfest.digitalocean.com/)\n[hacktoberfest.digitalocean.com](https://hacktoberfest.digitalocean.com/)\n\n[Build software better, together](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n\n[GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over…](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n[github.com](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n\n[OpenEBS](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n\n[OpenEBS is an open source storage platform that provides persistent and containerized block storage for DevOps and…](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n[openebs.io](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n\n- Thanks to [Satyam Zode](https://medium.com/@satyamz?source=post_page), [Akash Srivastava](https://medium.com/@srivastavaakash?source=post_page), [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page), and [OpenEBS](https://medium.com/@openebs?source=post_page).\n","notHasFeatureImage":false,"slug":"celebrate-hacktoberfest-2018-with-openebs-!!"},{"id":76,"title":"Keeping OpenEBS volumes in RW state during “Node down” scenarios","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"30-08-2018","tags":["Kubernetes","Chaos Engineering","Statefulset","OpenEBS"],"excerpt":"In this blog, I will go through a read-only issue faced at our lab in Kubernetes environment while using OpenEBS, and will also go through its possible workarounds.","content":"\nIn this post, I will go through a read-only issue experienced in the Kubernetes environment while using OpenEBS in our lab and possible workarounds.\n\nOpenEBS has the below deployment method for providing persistent storage to applications in Kubernetes clusters.\n\nAs shown in the above diagram, the application container runs on Node1, and an iSCSI target is runs on Node2 [There can be a case where the application and iSCSI target are running on the same node, but we don’t notice the issue in this scenario]. The iSCSI initiator of Node1 discovers and logs in to the iSCSI target and creates the disk `/dev/sdb`. The application consumes the mount point `/mnt/vol1` created over the disk `/dev/sdb` as persistent storage.\n\nWhen the node on which the iSCSI target, i.e. Node2, goes down, K8s then takes 5 minutes to schedule the iSCSI target on another node. This causes the mount point to enter a `Read-Only (RO)` state. Below are logs related to this issue:\n\n Aug 28 18:13:29 instance-1 kernel: [28477.898809] connection12:0: detected conn error (1020)\n Aug 28 18:15:30 instance-1 kernel: [28598.723742] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28598.730019] print_req_error: I/O error, dev sdb, sector 642376\n Aug 28 18:15:30 instance-1 kernel: [28598.737360] EXT4-fs warning (device sdb): ext4_end_bio:323: I/O error 10 writing to inode 12 (offset 186290176 size 258048 starting block 80360)\n Aug 28 18:15:30 instance-1 kernel: [28598.737364] buffer_io_error: 19448 callbacks suppressed\n Aug 28 18:15:30 instance-1 kernel: [28598.737365] Buffer I/O error on device sdb, logical block 80297\n Aug 28 18:15:30 instance-1 kernel: [28598.999756] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.006255] JBD2: Detected IO errors while flushing file data on sdb-8\n Aug 28 18:15:30 instance-1 kernel: [28599.006266] Aborting journal on device sdb-8.\n Aug 28 18:15:30 instance-1 kernel: [28599.012364] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.018784] Buffer I/O error on dev sdb, logical block 0, lost sync page write\n Aug 28 18:15:30 instance-1 kernel: [28599.028645] EXT4-fs error (device sdb): ext4_journal_check_start:61: Detected aborted journal\n Aug 28 18:15:30 instance-1 kernel: [28599.037759] EXT4-fs (sdb): Remounting filesystem read-only\n Aug 28 18:15:30 instance-1 kernel: [28599.045331] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.052005] Buffer I/O error on dev sdb, logical block 131072, lost sync page write\n\nAs shown in the above logs, when the connection is broken the iSCSI initiator reconnects and times out after 120 seconds. This makes the mount point go into RO mode. By this time, the application containers move into the “CrashLoopback” state. To bring applications back to the Running state, a lot of manual work must be done by remounting the mount point in RW mode. It is even possible that the application could treat the host directory as a mount point and will continue writing data, which can lead to data loss or corruption.\n\nIf the iSCSI initiator had waited for 300+ seconds, the mount should have stayed in RW mode, therefore reducing the amount of manual work.\n\nWe now need to find the setting from which the iSCSI initiator obtained this timeout value. Let us first look at “iscsiadm -m session -P 3” output.\n\n iSCSI Transport Class version 2.0–870\n version 2.0–873\n Target: iqn.2016–09.com.openebs.cstor:vol1 (non-flash)\n Current Portal: 10.142.0.2:3260,1\n Persistent Portal: 10.142.0.2:3260,1\n **********\n Interface:\n **********\n Iface Name: default\n Iface Transport: tcp\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 11\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 120\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n *****\n CHAP:\n *****\n username: \n password: ********\n username_in: \n password_in: ********\n ************************\n Negotiated iSCSI params:\n ************************\n HeaderDigest: None\n DataDigest: None\n MaxRecvDataSegmentLength: 262144\n MaxXmitDataSegmentLength: 262144\n FirstBurstLength: 262144\n MaxBurstLength: 1048576\n ImmediateData: Yes\n InitialR2T: No\n MaxOutstandingR2T: 1\n ************************\n Attached SCSI devices:\n ************************\n Host Number: 1 State: running\n scsi1 Channel 00 Id 0 Lun: 0\n Attached scsi disk sdb State: running\n \n\nTo set this value, iscsid.conf has a setting called “**node.session.timeo.replacement_timeout**”. Modifying this and performing a login into the iSCSI target produces the following output for the “iscsiadm -m session -P 3” command:\n\n <>\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 13\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 300\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n <>\n \n\nHere, we can see that “Recovery Timeout” under “Timeouts” section is now shown as 300.\n\nLet’s look at kernel logs when the connection is broken:\n\n Aug 28 18:38:14 instance-1 kernel: [29963.084882] connection13:0: detected conn error (1020)\n Aug 28 18:43:15 instance-1 kernel: [30263.588590] session13: session recovery timed out after 300 secs\n Aug 28 18:43:15 instance-1 kernel: [30263.596529] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:15 instance-1 kernel: [30263.602429] print_req_error: 16 callbacks suppressed\n Aug 28 18:43:15 instance-1 kernel: [30263.602430] print_req_error: I/O error, dev sdb, sector 643248\n Aug 28 18:43:15 instance-1 kernel: [30263.609938] EXT4-fs warning (device sdb): ext4_end_bio:323: I/O error 10 writing to inode 12 (offset 186736640 size 520192 starting block 80533)\n Aug 28 18:43:15 instance-1 kernel: [30263.609943] buffer_io_error: 3951 callbacks suppressed\n Aug 28 18:43:15 instance-1 kernel: [30263.609945] Buffer I/O error on device sdb, logical block 80406\n Aug 28 18:43:16 instance-1 kernel: [30265.229113] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:16 instance-1 kernel: [30265.236187] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:20 instance-1 kernel: [30269.488365] print_req_error: I/O error, dev sdb, sector 0\n Aug 28 18:43:20 instance-1 kernel: [30269.493883] print_req_error: I/O error, dev sdb, sector 0\n Aug 28 18:43:20 instance-1 kernel: [30269.499621] Buffer I/O error on dev sdb, logical block 0, lost sync page write\n Aug 28 18:43:20 instance-1 kernel: [30269.507654] EXT4-fs error (device sdb): ext4_journal_check_start:61: Detected aborted journal\n Aug 28 18:43:20 instance-1 kernel: [30269.517548] EXT4-fs (sdb): Remounting filesystem read-only\n Aug 28 18:43:20 instance-1 kernel: [30269.524898] EXT4-fs (sdb): previous I/O error to superblock detected\n Aug 28 18:43:21 instance-1 kernel: [30269.579913] sd 1:0:0:0: rejecting I/O to offline device\n \n\nPerfect!\n\nWe notice that the iSCSI initiator is actually using this timeout setting to stop the session recovery. Also, the mount point switched to read-only after 300 seconds.\n\n**What about the sessions that are already logged into iSCSI target?**\n\nThe below command can be used to change the setting for logged-in sessions:\n\n iscsiadm -m node -T -p -o update -n node.session.timeo.replacement_timeout -v 400\n\n“`iscsiadm -m session -P 3`” output is shown below:\n\n <>\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 14\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 400\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n ************************\n Attached SCSI devices:\n ************************\n Host Number: 1 State: running\n scsi1 Channel 00 Id 0 Lun: 0\n Attached scsi disk sdc State: running\n <>\n \n\nYou may notice the change in the “`Attached scsi disk`” value. This causes the volume to become unmounted and, therefore, needs to be remounted.\n\nIf you are aware of the iSCSI target login process, you likely know that it is a two-step process. The first step is to discover the target, and the second step is to log into the target. The “iscsiadm -o update” command can also be used after discovering the target but before logging into the target. Updating the setting in this way means that you do not need to remount the volume, as the login didn’t happen before the “iscsiadm -o update” command.\n\nBelow are the kernel logs related to the case where the iSCSI target is brought down and brought back before 400 seconds:\n\n Aug 29 11:31:37 instance-1 kernel: [19293.450614] connection12:0: detected conn error (1020)\n Aug 29 11:31:38 instance-1 iscsid: Kernel reported iSCSI connection 12:0 error (1020 — ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)\n Aug 29 11:31:41 instance-1 iscsid: connect to 10.142.0.2:3260 failed (Connection refused)\n Aug 29 11:32:26 instance-1 iscsid: message repeated 12 times: [ connect to 10.142.0.2:3260 failed (Connection refused)]\n Aug 29 11:32:29 instance-1 iscsid: connect to 10.142.0.2:3260 failed (Connection refused)\n Aug 29 11:32:37 instance-1 iscsid: message repeated 2 times: [ connect to 10.142.0.2:3260 failed (Connection refused)]\n Aug 29 11:36:38 instance-1 iscsid: connection12:0 is operational after recovery (79 attempts)\n\nSuperb!!!\n\nAs you can see in the above logs, the iSCSI connection was successful even after 300 seconds. The mount point did not go into RO state, and thus the application container will remain in the `Running` state. This avoids a lot of manual work for the user.\n\n**What about cases when the iSCSI login is already done, and the volume mountpoint can’t be remounted?**\n\nOne way to do this is by modifying the content of a file in the `/sys/class/iscsi_session/` directory path, from which the iSCSI initiator reads this setting.\n\nThe file related to this setting is as follows:\n\n /sys/class/iscsi_session/session/recovery_tmo\n\n“iscsiadm -m session -P 3” provides the session ID related to iSCSI login. It will be the “SID” parameter under the “Interface” section. For the previous output, the file to change is:\n\n /sys/class/iscsi_session/session14/recovery_tmo\n\nExecute this command to run as a root to modify this setting within the file:\n\n echo 400 > /sys/class/iscsi_session/session/recovery_tmo\n\n**Conclusion**: When Kubernetes takes time in rescheduling the iSCSI target pod into a different node, modifying this setting in any of the above methods prevents the mount point from entering the RO state.\n\n**There is one “gotcha”**: If multipathing is enabled, the multipath-related setting takes precedence over this setting if the setting modification is done either through the iscsid.conf file or “iscsiadm -o output” command.\n","notHasFeatureImage":false,"slug":"keeping-openebs-volumes-in-rw-state-during-“node-down”-scenarios"},{"id":77,"title":"OpenEBS 0.7 Release pushes cStor Storage Engine to field trials!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-08-2018","tags":["Container","Kubernetes","OpenEBS","Storage Solutions"],"excerpt":"Before I get into some fascinating features of 0.7 release, I would like to take this opportunity to thank all users who have taken our OpenEBS survey and have come forward to share your Kubernetes Stateful Workload and OpenEBS adoption stories.","content":"\nBefore I get into some fascinating features of 0.7 release, I would like to take this opportunity to thank all users who have taken our OpenEBS survey and have come forward to share your Kubernetes Stateful Workload and OpenEBS adoption stories. When I hear statements like ***“OpenEBS just works!”***, it is definitely a sign that the project and the OpenEBS Developer Community are headed in the right direction. It is both heartening and humbling. Thank you!\n![contributions survey](https://cdn-images-1.medium.com/max/800/08J2dBBRWCnbu82R_)\nWe would welcome more contributions to the survey mentioned above, for example, whether you are an OpenEBS user (yet) or not: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\n—\n\nWhile 0.6 was mainly focused towards making the storage resilient via building chaos tools like [Litmus](https://github.com/openebs/litmus); 0.7 has taken a huge leap towards bringing a new Storage Engine for your OpenEBS Volumes — called cStor. — **container** **Storage** and incidentally it is a **Storage** written in **c**)!\n\nSpeaking of OpenEBS 0.7, the geek in me would like to banter about the OpenEBS CAST (Container Attached Storage Templates or read it as moulding) framework that we developed to help easily plugin new Storage Engines, or about how we use the new Kubernetes Dynamic Client to do some heavy lifting of interacting with the Kubernetes CRDs and Scheduling. Anyways, let us keep it for the upcoming KubeCons!\n\nSo, here are some things that make OpenEBS 0.7 a significant milestone:\n\n- Increased number of Integrations\n- Node Disk Manager for Kubernetes\n- OpenEBS Control Plane\n- cStor Storage Engine\n\nIn the rest of this post, I will provide additional details on these items.\n\n—\n\n## Integrated Offerings:\n\nOpenEBS remains committed to being Easy to Setup and Use, and we are continuing our efforts to make it available from your Cluster Solutions such as KubeApps, IBM Private Cloud, StackPoint Cloud, Red Hat OpenShift, Rancher, Kontena Pharos, etc.\n\n—\n\n[Node Disk Manager (NDM)](https://github.com/openebs/node-disk-manager) for Kubernetes has made its first [release](https://github.com/openebs/node-disk-manager/releases)! It helps you better manage the Disk Inventory in your Kubernetes Cluster. The discovered block devices are added as Kubernetes custom resources called Disks. In addition to managing actual disks attached to nodes, NDM also helps developers simulate disks on their local machines using sparse files. *You can interact with NDM via kubectl.*\n\n kiran_mova@kmova-dev:~$ kubectl get disks --show-labels\n NAME AGE LABELS\n disk-cdf294e8662a87.. 2h kubernetes.io/hostname=node1...\n disk-eaf3931c96c1d9.. 2h kubernetes.io/hostname=node2...\n disk-edc04b8fcf924e.. 2h kubernetes.io/hostname=gke-kmova-\n\n To learn more about NDM and its commands, please visit\n https://github.com/openebs/node-disk-manager\n\n—\n\n## OpenEBS Control Plane (maya):\n\nAs you use OpenEBS, you realize that you almost never interact with any of the actual components of OpenEBS. This is thanks in part to the beauty of Kubernetes and how OpenEBS plugs itself into the Kubernetes orchestration layer. You can hardly notice when there is an addition to the control plane that helps provision and manage OpenEBS Volumes.\n\nTypical Storage Solutions come with a separate heavy-weight control plane, which essentially must do everything that Kubernetes does, albeit for Storage constructs like Pools and Volumes. If Storage Volumes and Pools are treated as Containers, then Kubernetes itself can Orchestrate Storage, right?\n\nThis is exactly what OpenEBS does; the Storage Volumes and Pools themselves are represented by Containers (Deployments and Services), and Kubernetes does most of the Orchestration! What is really needed by OpenEBS is to produce the Kubernetes YAMLs with the appropriate resources and scheduling constructs.\n\nThe way the Storage Pods are deployed can have specific requirements such as:\n\n- Pods should be deployed as StatefulSet or DaemonSet or regular Deployment with the custom Scheduler.\n- Pods should be associated with Node Selectors or Taints/Tolerations etc.\n- Pods should have specific privileges and/or resource limits, etc.\n- Pods should be configured with Affinity and Anti-affinity rules that make the storage highly available across node or zone failures etc.\n\nOpenEBS Release 0.7 makes it insanely easy to specify such constructs and associate them with Storage Engines, basically by just adding a few YAML files.\n\nTo understand this better, let’s observe a couple of examples.\n\n**Example 1:** Scheduling PVs of Mongo instances across Availability Zones. When Stateful Applications like Mongo are deployed, each Mongo instance gets its own PV and PVC. To be truly resilient against node failures, the PVs (actual storage) belonging to a given Mongo application must be spread out across nodes or occasionally even across availability zones. We already know how to ensure Mongo instances (pods) are spread out with Pod Anti-affinity rules.\n\nWith OpenEBS, the same Pod Anti-affinity rules can be passed on and the OpenEBS Control Plane (maya) takes care of setting them on the storage pods. All you need to do is the following:\n\n**Step 1:** Create an OpenEBS StorageClass that can be used by the PVs of Mongo StatefulSet:\n\n # Create a StorageClass suited for Mongo STS\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mongo-pv-az\n annotations:\n cas.openebs.io/config: |\n - name: ReplicaCount\n value: \"1\"\n - name: ReplicaAntiAffinityTopoKey\n value: failure-domain.beta.kubernetes.io/zone\n - name: StoragePool\n value: default\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/fstype: \"xfs\"\n\n**Step 2:** Specify the application label (say *app-unique-id*) in the Mongo StatefulSet:\n\n apiVersion: apps/v1beta1\n kind: StatefulSet\n metadata:\n name: mongo\n spec:\n serviceName: \"mongo\"\n replicas: 3\n template:\n metadata:\n labels:\n role: mongo\n environment: test\n openebs.io/replica-anti-affinity: app-unique-id\n\n**Step 3:** Specify the Storage Class in the Volume Claim Template:\n\n volumeClaimTemplates:\n — metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: mongo-pv-az\n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nWhen the PVs are created for the Mongo StatefulSet, the Replicas are configured with PodAntiAffinity rules with the provided application label and topology key as follows:\n\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n — labelSelector:\n matchLabels:\n openebs.io/replica: jiva-replica\n openebs.io/replica-anti-affinity: app-unique-id\n topologyKey: failure-domain.beta.kubernetes.io/zone\n\nThe complete YAML file for Mongo StatefulSet using OpenEBS PV is available [here](https://github.com/openebs/openebs/blob/master/k8s/demo/mongodb/mongo-statefulset.yml).\n\n**Example 2:** When running in hyper-converged mode along with applications, it is imperative to deploy granular control on how much memory or CPU can be allocated for serving a storage volume. Again, this is a relatively common scenario, and we know how to control it for the Application Deployments! The same can be extended to the Storage!\n\nFor example, you can define the Resource Requests and Limits for different storage pods:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-jiva-limits\n annotations:\n cas.openebs.io/config: |\n - name: TargetResourceRequests\n value: |-\n memory: 0.5Gi\n cpu: 100m\n - name: TargetResourceLimits\n value: |-\n memory: 1Gi\n - name: ReplicaResourceRequests\n value: |-\n memory: 0.5Gi\n cpu: 100m\n - name: ReplicaResourceLimits\n value: |-\n memory: 2Gi\n cpu: 200m\n\nI have only covered a couple of examples here, but I am very excited that with the framework we have now embedded within OpenEBS, we can support any kind of scheduling requests you might have. **Bring it On!**\n\n—\n\n**cStor Storage Engine (cStor)** is the latest addition to the storage engines supported by OpenEBS. Similar to the previous Jiva/Longhorn storage engine, cStor is also completely Open Source and has no dependency on the Kernel. It can run in the platform of your choice — either on-premise or cloud!\n\nWhile cStor Volumes work just like the previous OpenEBS volumes in terms of replication and high availability, cStor has many advantages:\n\n1. Managing the Storage Space — You no longer need to worry about writing extra scripts for cleanup when Volumes are deleted.\n2. cStor reduces the number of Containers required to achieve replication capability. This is a significant shift from requiring 4 pods per Volume to just 1 new pod per Volume. The Replication is taken care of by a “Pool” pod that can serve multiple volumes.\n3. The core storage engine used to store the data is based on OpenZFS technology that has been used in production for over a decade now.\n\ncStor Pools are created on the Disks discovered by NDM. (Again, the geek in me would like to point out the excellent Operator pattern used to convert a Storage Pool Claim to Storage Pools and the cool way in which we used side-cars and CRDs to manage a non-Kubernetes storage engine in a Kubernetes way!)\n\ncStor is available as an Alpha feature with 0.7 and is very easy to try out! You can read more about cStor [here](/docs/concepts/cstor?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294#cstor).\n\nThere is a lot that I didn’t cover or address in this blog post, but I hope this served as a brief introduction to some of the most exciting features in the 0.7 release. For detailed release notes and instructions for getting started, visit [release notes](https://github.com/openebs/openebs/releases/tag/v0.7).\n\nIn my opinion, OpenEBS 0.7 decidedly will be remembered as a release that brought the coming of age of several different OpenEBS projects like [Litmus](https://github.com/openebs/litmus), [Node Disk Manager for Kubernetes (NDM)](https://github.com/openebs/node-disk-manager) and [OpenEBS cStor](/docs/concepts/cstor?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294). OpenEBS 0.7 is equipped to successfully deliver a comprehensive storage solution for DevOps persona, providing capabilities for automating Storage management in their Kubernetes Clusters.\n\nWe would welcome more contributions to the survey mentioned above, whether you are an OpenEBS user (yet) or not: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\nWhat makes this project great is your support and feedback! Please reach out to us on Slack or comment below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294)/\n\nThanks to [Amit Das](https://medium.com/@amit.das?source=post_page) and [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page). [Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":true,"slug":"openebs-0.7-release-pushes-cstor-storage-engine-to-field-trials!"},{"id":78,"title":"Provisioning Kubernetes cluster by kubeadm","author":"Chandan Kumar","author_info":"Software Engineer at MayaData Inc","date":"22-08-2018","tags":["Containerized Storage","Kubeadm","Kubernetes","OpenEBS","Stateful Workloads"],"excerpt":"Kubeadm is a new tool that is part of the Kubernetes distribution of 1.4.0. It allows you to install and set up a Kubernetes cluster.","content":"\n**Kubeadm** is a new tool that is part of the Kubernetes distribution of 1.4.0. It allows you to install and set up a Kubernetes cluster. One of the most frequent criticisms of Kubernetes is that it’s difficult to install. Kubeadm makes this much easier, so I strongly suggest you give it a try.\n\n### Pre-requisites for creating a cluster:\n\n- One or more machines running the compatible OS (ex: Ubuntu)\n- 2-GB or more of RAM per machine\n- 2-CPU or more for Master\n\nNetwork connectivity among all machines in the cluster\n\n### Objectives:\n\n- Install a single master Kubernetes cluster.\n- Install a Pod network on the cluster so that your Pods can communicate.\n\nBeginners can set up the pre-requisites in their own machine by creating virtual machines (VMs) in a virtual box, or they can also use multiple machines for creating clusters.\n\n### Installation:\n\nInstall these requirements in each node:\n\n- Docker\n```\n $ sudo apt-get update\n $ sudo apt-get install -y docker.io\n``` \n\n- Kubeadm, Kubelet, Kubectl\n```\n $ sudo apt-get update && sudo apt-get install -y apt-transport-https curl\n $ sudo -i\n $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\n $ cat </etc/apt/sources.list.d/kubernetes.list\n $ deb http://apt.kubernetes.io/ kubernetes-xenial main\n $ EOF\n $ exit\n $ sudo apt-get update\n $ sudo apt-get install -y kubelet kubeadm kubectl\n``` \n\n### Master Node:\n\nThe master is the machine where the control plane components run, including etcd (the cluster database) and the API server (which the kubectl CLI communicates with).\n![Master Node](https://lh5.googleusercontent.com/BbzeYd9ttUgNLHIn-zS1gndo_sCNLYOiqe-HTHQLDNqegK72Lc7Nzg88tfUCXSo_p6Wyrq-beic2mf0ZR7tFMw3dW_IZvGYC-MT-GVCSWPqoS4OsglbwCQ2ZskBuT2FsJLI9jzY-UdokGUJnpA)\nBefore running kubeadm init in master node, first, choose a pod network add-on and verify whether it requires any arguments to be passed for kubeadm initialization. Depending on which third-party provider you choose, you might need to set the --pod-network-cidr argument with kubeadm init .\n\n- [list of pod network add-on.](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network)\n- [list of arguments](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/)\n\n### Configure the cgroup Driver used by kubelet\n```\n $ sudo sed -i \"s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g\" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n```\n### Restart kubelet\n```\n $ sudo systemctl daemon-reload\n $ sudo systemctl restart kubelet\n``` \n\nExample:\n\n```\n $ sudo kubeadm init --apiserver-advertise-address= --apiserver-cert-extra-sans=10.0.2.15 --pod-network-cidr 10.1.0.0/16\n $ sudo mkdir -p $HOME/.kube\n $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n $ sudo chown $(id -u):$(id -g) $HOME/.kube/config\n $ sudo sysctl net.bridge.bridge-nf-call-iptables=1\n $ sudo KUBECONFIG=/etc/kubernetes/admin.conf\n $ kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml\n```\n\nAfter you finish running kubeadm init in master node, it provides the token, master-ip, sha and hash as follows:\n```\n $ kubeadm join --token : --discovery-token-ca-cert-hash sha256:\n```\nIf you do not have the token, you can obtain it by running the following command on the master node:\n```\n $ kubeadm token list\n```\nBy default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token using the following command:\n```\n $ kubeadm token create\n```\nFor reference, you can view this document: [https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)\n\n### Worker nodes:\n\nA worker node in Kubernetes was previously known as a minion. A node may be a VM or a physical machine, depending on the cluster. Each node has the services necessary to run pods and is managed by the master components.\n\n### Joining worker nodes:\n\nTo add nodes to your cluster, do the following for each machine:\n\n- SSH to the machine\n- Become root (e.g. sudo su -)\n- Run the command that was returned by kubeadm init. For example:\n\n $ kubeadm join — token : — discovery-token-ca-cert-hash sha256:\n\nNow you are all set and can list the nodes from the master by running\n```\n $ kubectl get nodes\n```\nPlease leave your valuable comments and questions below.\n","notHasFeatureImage":true,"slug":"provisioning-kubernetes-cluster-by-kubeadm"},{"id":79,"title":"Building persistent storage with OpenEBS and Local SSDs on GKE","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"21-08-2018","tags":["OpenEBS","Solutions","Stateful Applications"],"excerpt":"In this blog post, we’ll discuss the advantages of using GKE Local SSDs instead of GPDs and how to set up persistent storage using OpenEBS for highly available storage volumes.","content":"\nIf you are currently using Google Cloud for your Kubernetes needs, it is more than likely that you are using Google’s Kubernetes service, GKE. The good news is that you have another option for persistent storage apart from Google Persistent Disks, or GPDs.\n\n> *With OpenEBS software, you can easily set up persistent storage using the Local SSDs of GKE instances.*\n\nIn this blog post, we’ll discuss the advantages of using GKE Local SSDs instead of GPDs and how to set up persistent storage using OpenEBS for highly available storage volumes. Though GPDs are extremely easy to provision and natively integrated with GKE, there are some inconveniences with GPDs related to stateful applications.\n\n## Running Stateful Apps using GPD Volumes\n![GPDs as persistent disks may not be readily available](https://lh5.googleusercontent.com/JOkShLc8RiH5a-bWvFPqNWn9rs24C3toodcI3i_g7vew2-4FzYdL_-vX7X9nofXCwTMU0P9zVraY6NSRCIE5jfdlrIeXXSW7KuT74dPv7QnGn6oVRdbAKN1U9GkR-UdzDyOTPMqvoxX1EoNzIQ)\n\nWhen a GKE node goes down, a new node comes up as part of Cluster Autoscaler. GPD disks that are associated with the old node must be detached from the old node and attached to the new node. This must be done through a manual procedure. Even if automated detaching were available, it would still slow the system.\n\n- Performance of the GPD volumes is limited as they are part of a larger shared storage network. GKE is not able to make use of faster disks such as SSDs.\n- Slow fail-over essentially means no High Availability.\n- Poor I/O, unless unused disk space is not an issue.\n\n## Restrictions for Using Local SSDs as -is for Kubernetes\n\nGoogle cloud documentation on the restrictions for using local SSDs for Kubernetes is mentioned [here](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd).\n\n![Use of local disks has restrictions on GKE](https://lh4.googleusercontent.com/uE8p6fJtr2AYnqYAuv_4XWEf5ZQ1lgZ8p-wthognqxV2ayxGQf2iCh7C6LEK5qM_OwvrnDx3R4D9BaiQLac8LE1fmqm3u1BI_bDFYn2c3T-zc1cFmuvL-0WVZoAbHsv4Vo0elHttSxmdQv9OUw)\n### Here is a quick reference summarizing the restrictions.\n\nBecause local SSDs are physically attached to the node’s host virtual machine instance, any data stored exists only on that node. Since the data stored on the disks is local, your application must accommodate for this data being unavailable.\n\n[Data stored on local SSDs are ephemeral](https://cloud.google.com/compute/docs/disks/local-ssd#data_persistence). A Pod that writes to a local SSD might lose access to the data stored on the disk if the Pod is rescheduled away from that node. Additionally, if the node is terminated, upgraded, or repaired, the data will be erased.\n\nAdditionally, you cannot add local SSDs to an existing node pool.\n\n### No Restrictions with OpenEBS on the Local SSDs of GKE\n\nOpenEBS natively supports synchronous replication. This replication can be done when nodes (hence disks) are placed across Availability Zones (AZs).\n\n### Synchronous Replication of Data with OpenEBS\n\nOpenEBS uses a minimum of three replicas to run OpenEBS clusters with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk, which will come up as part of Cluster Autoscaler. In the meantime, your workload stems from accessing the live data from one of the replicas.\n![](https://lh3.googleusercontent.com/EAMk9eGAPvqcTPAd8IH_VWcfzNaSyMGEVhDDYj8CKvYqAtXc1JvmQHw_CeD-Qc_4Ny3Ahav4LjLhZgiEekcuSnij1BD2tqOEJZI-esCPP9wVsKXEiZMt-DsML0lRs6Dm4K-7LGr_XQ91IAdRfw)\n### Quick Steps to Set Up OpenEBS on GKE\n\n- Set up a three-node GKE cluster with local disks by enabling the Cluster Autoscaler feature.\n- Install OpenEBS on Kubernetes Nodes. This should be simple, and a couple of methods are discussed at the beginning of our docs, using either a Helm Chart or directly from Kubectl. More details are mentioned in the [OpenEBS documentation](/docs).\n- Use OpenEBS Storage Classes to create Persistent Volumes for your stateful applications.\n\n## Detailed Explanation of OpenEBS 0.7 Cluster Deployment on GKE across AZs and Rebuilding of PVs.\n\n### Prerequisites\n\n- GKE account with credentials to create instances using the local disk.\n- Ubuntu 16.04\n- gcloud CLI access\n\n### Creating a GKE Cluster\n\nYou can start by entering your login information to your Google Cloud Platform and create the GKE cluster as follows:\n\n1. Click “CREATE CLUSTER” from the console and fill in the required details.\n\n![](https://lh3.googleusercontent.com/jjAzX7Vykgt0ChGM5MdQ8K6cEdxiB9-3xWO309T4hSizm__EnWsH_pO61lgJZQAbDEJim9K0OmfKYu7IgqhUg_BhYw94YinMDbA72GvkoC7Vmzzgrw4q0rqYfOSbNqIV-F16ONCkJa1lN0e6gw)\n2.  Click “Advanced Edit” under the Node Pool section and Enable the “Auto Scaling” feature for the cluster. Select the Node Image type as “Ubuntu.”\n![](https://lh3.googleusercontent.com/ud5ky3UMDH8GL1FuOLrXSaVC-9q1-3bdYjEPzHcPM_uhWc_h8NIGa1s2yKuUHhAe1dLo388jGupB8mVcYfBIDo9zS0ABqOzU9YxclXvaoHTtXprtAjArXUNh4IoEgirTw21wiMCIKAyQZCGZQQ)\n3.  Add the required number of local SSD disks per Node as follows and save the changes.\n![](https://lh4.googleusercontent.com/6PJNdbLybtKYi7vQrBHjCmoH8AjYNFh-FaMXX0AEpm0KyFfsL3lwTHMuF1H1s0P8UasWmC7kk2_83iJXYM0_mHYfJ2HSwo58iGAbOoP94an6GdlJwjAIYTmqdza0Bj-IknbZaSraTokLtaTGIA)\n4. We have deployed the cluster across different AZs. For high availability, and in case one zone goes down, the data and application would remain intact. You can select the required zones by clicking on “Advanced options”. I have chosen the following zones rather than my default zone.\n![](https://lh3.googleusercontent.com/n8sUfShxkqLsL5btIo1RucZftcalGmbbZNxtaNqhjhT4CxkeowKdY1yhB-BF5TbBWHfnqMo3YAWWuhCGwpBfXZLq2m00rAeEshnzvOSEp0Xz3Z9k0qXOacurtXL8HDDgnAnd5_BRWFVYSkjNEA)\n5. Now you can create your K8s cluster by clicking the “Create” button.\n\n6. Check your Kubernetes Cluster status under the “Kubernetes Clusters” section.\n\n7. Once your cluster is ready, launch the gcloud shell to access your cluster using CLI.\n\n8. You may need an admin-context if you are installing OpenEBS without Helm. Use the following command to set the admin-context:\n ```\n kubectl create clusterrolebinding -cluster-admin-binding --clusterrole=cluster-admin --user=\n ```\n9. Before installing OpenEBS, you will need to select one of the mounted SSD disks to create an OpenEBS Storage Pool. You must note the mount path of the selected disk, as we need to provide the mount path details while creating the storage pool.\n\nIn my cases, I have chosen two local SSDs per node, and it is mounted automatically on the Node as part of the instance creation. Among these disks, I will choose the “sdb” disk. The OpenEBS Jiva storage pool will be created on the “sdb” disk on each node. Steps for creating pools are mentioned below.\n\n You can obtain the mounted disk details on Nodes using the following command:\n\n sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL\n\n Below is an example output.\n\n NAME FSTYPE SIZE MOUNTPOINT LABEL\n sdb ext4 375G /mnt/disks/ssd0\n sdc ext4 375G /mnt/disks/ssd1\n sda 100G\n └─sda1 ext4 100G / cloudimg-rootfs\n\n10. You are now ready to install the OpenEBS 0.7 cluster using the following command. Then follow the steps below from your master node.\n\n ```\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.7.0.yaml\n\n ```\n\n11. You can check the OpenEBS running pod details using the following command:\n\n kubectl get pods -n openebs\n\nBelow is an example output:\n\n NAME READY STATUS RESTARTS AGE\n cstor-sparse-pool-9stx-5b6556cc54–4q249 2/2 Running 0 22s\n cstor-sparse-pool-ew8c-54bd645b6c-6s2d5 2/2 Running 0 22s\n cstor-sparse-pool-fml8–8685fbbc74-dg4fs 2/2 Running 0 22s\n maya-apiserver-85657db5cf-b7sr9 1/1 Running 0 1m\n openebs-ndm-857lr 1/1 Running 0 57s\n openebs-ndm-gpxh6 1/1 Running 0 57s\n openebs-ndm-tjmhg 1/1 Running 0 57s\n openebs-provisioner-7594dbf8cd-ph8jn 1/1 Running 0 59s\n openebs-snapshot-operator-7dddbf56db-7xtct 2/2 Running 0 58s\n \n\n12. You can then check the default storage classes created as part of the OpenEBS operator installation using the following command:\n\n ```\n kubectl get sc\n ```\nAgain, here is an example output:\n\n NAME PROVISIONER AGE\n openebs-cstor-sparse openebs.io/provisioner-iscsi 1h\n openebs-jiva-default openebs.io/provisioner-iscsi 1h\n openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 1h\n standard (default) kubernetes.io/gce-pd 1h\n \n\n13. Create a storage pool on an external disk mounted on the nodes. In step 8, I mentioned using the “sdb” disk for creating an OpenEBS storage Pool. To create a storage pool, create a file called “openebs-config.yaml” in your master node and add the example YAML (shown below) while changing the appropriate mounted disk path.\n\nFor example, if your external disk is mounted as `/mnt/disks/ssd0` in your nodes, change the path as shown below.\n\n path: “/mnt/disks/ssd0”\n\n Example yaml file:\n\n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: default\n type: hostdir\n spec:\n path: \"/mnt/disks/ssd0\"\n ---\n \n14. Apply the modified *openebs-config.yaml* file by using the following command:\n\n ```\n kubectl apply -f openebs-config.yaml\n ```\n\nThis will create a storage pool called “default” on the selected disk.\n\n15. The storage pool is now created on the Nodes according to your requirement. You can now get the storage pool details by running the following command:\n\n ```\n kubectl get sp\n ```\n\n Example output:\n\n NAME AGE\n cstor-sparse-pool-9stx 1h\n cstor-sparse-pool-ew8c 1h\n cstor-sparse-pool-fml8 1h\n default 1h\n \n\n16. Now your configured OpenEBS Jiva storage engine will create an OpenEBS Jiva volume on the storage pool established on your local SSD disk.\n\n17. Deploy your application YAML, which will be consumed in the OpenEBS Jiva volume.\n\nWe now have the Percona deployment application YAML and need to change the storage class mentioned in the YAML with the default storage class name “openebs-jiva-default” for the Jiva volume. This will be created as part of the *openebs-operator.yaml* installation.\n\n18. We now use the following Percona deployment YAML file to deploy a stateful Percona application using the Jiva volume. Obtain the YAML file using following command:\n```\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-deployment.yaml\n```\n\n19. Edit the downloaded “percona-openebs-deployment.yaml” and conduct the following changes.\n\nIn the “PersistentVolumeClaim” section, under metadata, add the following content.\n\n labels:\n \n “volumeprovisioner.mapi.openebs.io/replica-topology-key-domain”: “failure-domain.beta.kubernetes.io” \n “volumeprovisioner.mapi.openebs.io/replica-topology-key-type”: “zone” \n\nNow change the “storageClassName” from ”openebs-standard” to “openebs-jiva-default”.\n\n20. Save the modified changes and apply the YAML as follows. This will create a PVC and PV in the mentioned size.\n\n\n kubectl apply -f percona-openebs-deployment.yaml\n\n\n21. You can then view the PVC status by running the following command:\n\n kubectl get pvc\n\n Below is an example output.\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n \n demo-vol1-claim Bound default-demo-vol1-claim-2300073071 5G RWO openebs-jiva-default 11m\n\n\n22. You can then obtain the PV status by running the following command:\n\n kubectl get pv\n\nAnd here is an example output.\n\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n default-demo-vol1-claim-2300073071 5G RWO Delete Bound default/demo-vol1-claim openebs-jiva-default\n \n\n23. Now, your Percona application pod will be running along with three Jiva volume replicas and one Jiva Controller pod. You can view the running pod status by running the following command:\n\n\n kubectl get pods -o wide\n\n\nWhich will provide the following output.\n\n NAME READY STATUS RESTARTS AGE IP NODE\n default-demo-vol1-claim-2300073071-ctrl-78d4fbc66d-v4454 2/2 Running 0 14m 10.44.2.9 gke-mayadata-gke-default-pool-2d42a511-7gcz\n default-demo-vol1-claim-2300073071-rep-79f97d5494-cvd8z 1/1 Running 0 14m 10.44.0.8 gke-mayadata-gke-default-pool-5eed8925-7391\n default-demo-vol1-claim-2300073071-rep-79f97d5494-g2hff 1/1 Running 0 14m 10.44.1.8 gke-mayadata-gke-default-pool-54c3be93-qsgl\n default-demo-vol1-claim-2300073071-rep-79f97d5494-qqvv4 1/1 Running 0 14m 10.44.2.10 gke-mayadata-gke-default-pool-2d42a511-7gcz\n percona-7f6bff67f6-mjp9d 1/1 Running 0 14m 10.44.1.9 gke-mayadata-gke-default-pool-54c3be93-qsgl\n \n\n### Summary\n\n![OpenEBS on GKE](https://lh5.googleusercontent.com/3zgM48Nep0Uszwxu_mIfewSP2SXCSdWHTeSzA7JbJfgaQnbaxB8SVd5gF6ADpXmmnnd84NqctMrm7CfFlkDuhSTPsuLCWxRm9dgtSXI6bOh-Gl_oKsbkubON_To3QcDbHXJA20n909Jie0lLXQ)\n> *Local SSDs on your GKE cluster can be used as persistent storage for stateful applications such as Prometheus, WordPress, MongoDB, etc. This gives you the advantages of both the low latency local SSD use and fault-tolerant architecture ensured by OpenEBS.Thank you for reading. Feel free to join our *[*Slack*](https://slack.openebs.io/)* channel for any questions or help needed, or to share your success stories of using OpenEBS on GKE.*\n","notHasFeatureImage":false,"slug":"building-persistent-storage-with-openebs-and-local-ssds-on-gke"},{"id":80,"title":"cStor Pool Provisioning in OpenEBS 0.7","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"19-08-2018","tags":["Kubernetes","OpenEBS","Pool","Storage"],"excerpt":"OpenEBS team is happy to announce the release of 0.7 which comes with a new storage engine for creating storage pool known as cStor engine.","content":"\nGreetings OpenEBS users!\n\nThe OpenEBS team is happy to announce the release of 0.7, which comes with a new storage engine, known as cStor, for creating storage pools.\nTo find out more details on this specific release, please visit the following links:\n[https://github.com/openebs/openebs/releases](https://github.com/openebs/openebs/releases)\n[https://openebs.io/blog/openebs-07-release-pushes-cstor-storage-engine-to-field-trials](/blog/openebs-07-release-pushes-cstor-storage-engine-to-field-trials)\n\nTo keep this story short and concise I will jump directly into how you can provision a storage pool in 0.7 using the cStor engine. Just for your information, storage pools can also be provisioned using the Jiva engine that we used in previous versions of OpenEBS.\n\nLet’s get started!\n\nThere are currently two ways to provision a storage pool using the cStor engine in OpenEBS 0.7. As we move forward with the tutorial, I will assume that you already have a Kubernetes cluster set up. I will follow up this tutorial by having a 3-node Kubernetes cluster with one physical disk attached to every node on GKE.\n\n### Manual Pool Provisioning\n\nTo provision a storage pool manually, you run the following command where your kubernetes cluster is configured.\n\n ashutosh@miracle:~$ kubectl get disk\n NAME CREATED AT\n disk-26ac8d634b31ba497a9fa72ae57d6a24 1d\n disk-2709a1cba9cea9407b92bc1f7d1a1bde 1d\n disk-427145375f85e8a488eeb8bbfae45118 1d\n sparse-4b488677f76c94d681870379168a677a 1d\n sparse-c3ddc8f0de2eb17c50d145cf6713588c 1d\n sparse-e09fe4b5170a7b8fd6b8aabf8c828072 1d\n\nThe output with prefix disk represents your physical disks, and the output with prefix sparse represents sparse disks. We will discuss the sparse disk concept in a later blog-post, but for now, let’s concentrate on physical disks! We need to simply copy the physical disks (in the diskList field of the storage pool claim YAML) over which the pool should be created. The SPC YAML will look like the following:\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk \n poolSpec:\n poolType: striped\n disks:\n diskList:\n - disk-26ac8d634b31ba497a9fa72ae57d6a24\n - disk-2709a1cba9cea9407b92bc1f7d1a1bde\n - disk-427145375f85e8a488eeb8bbfae45118\n\nNow you just need to apply the created YAML. That’s it. Done!\n\n ashutosh@miracle:~$ kubectl apply -f spc.yaml \n storagepoolclaim.openebs.io/cstor-disk created\n ashutosh@miracle:~$ kubectl get sp\n NAME CREATED AT\n cstor-disk-5wsi 4s\n cstor-disk-7pgs 4s\n cstor-disk-8fhi 4s\n default 14d\n\nEach disk that we entered in SPC YAML is attached to a specific node. So, 3 cStor pools were created on top of the 3 nodes, and this sp (the output belongs to the applied SPC) belongs to the applied SPC. If all 3 disks were attached to a single node, we would only have one sp.\n\n### Dynamic Pool Provisioning\n\nThe steps listed above involved a heavy manual process, but it helps the user to configure a storage pool based on their choices. If you didn’t like that manual process, don’t worry; let’s work some magic here by doing something known as dynamic pool provisioning.\n\nStart by applying the following SPC YAML :\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-dynamic\n spec:\n name: cstor-disk-dynamic\n type: disk\n # required in case of dynamic provisioning\n maxPools: 3 \n # If not provided, defaults to 1 (recommended but not required)\n minPools: 3\n poolSpec:\n poolType: striped\n ashutosh@miracle:~$ kubectl apply -f dynamic_spc.yaml storagepoolclaim.openebs.io/cstor-disk-dynamic created\n ashutosh@miracle:~$ kubectl get sp\n NAME CREATED AT\n cstor-disk-dynamic-jwc5 6s\n cstor-disk-dynamic-qot0 6s\n cstor-disk-dynamic-s8va 6s\n default 14d\n\nThat’s it. Done!\n\nLet’s better understand what’s going on here:\n\n1. The dynamic way of pool provisioning will support reconciliation, i.e. the OpenEBS control plane will always try to get the maxPools number of pools specified on storagepoolclaim. If the SPC YAML is applied and the node is down or there are no existing disks, when the resources come up, the pool will be provisioned automatically without any intervention.\n2. The manual method of provisioning will not have any such reconciliation.\n3. The number of the pool(s) specified by minPools will be created or no pools will be provisioned. So, if maxPool=10 and minPool=6, the Control plane will always try to get to a pool count of 10 but any single shot of provisioning in any one part of the reconciliation loop must provision at least 6 pools. Once the minPools count is reached even if the count of pool increases by only 1, the control plane will execute the action.\n\n### NOTES: \n\n1. In the above tutorial, we provisioned a striped type of pool for both cases. A mirrored pool can also be provisioned, but at least 2 disks must be attached to the node. Simply change the poolType field in SPC YAML to mirrored.\n2. The number of sp resources created is equal to the number of cStor pools created on top of each node, but those belong to a single SPC that spawns them. For applications, the pool is virtually one while we create volumes.\n\nHopefully, this helps! Feel free to post any queries or concerns and share any feedback. You can reach out on our OpenEBS slack channel as well.\n([https://slack.openebs.io/](https://slack.openebs.io/))\n","notHasFeatureImage":false,"slug":"cstor-pool-provisioning-in-openebs-0.7"},{"id":81,"title":"How to start contributing to mayactl","author":"Sumit Lalwani","author_info":"Sumit Lalwani is a Software Engineer at Mayadata. He is a Kubernetes enthusiast and passionate about open source, containers, cloud, and arm. He loves to learn and code.","date":"14-08-2018","tags":["Docker","OpenEBS","Kubernetes"],"excerpt":"mayactl is the command line tool for interacting with OpenEBS volumes. mayactl is not used/required while provisioning or managing the OpenEBS volumes, but it is currently used while debugging and troubleshooting.","content":"\n## What is mayactl?\n\n- mayactl is the command line tool for interacting with OpenEBS volumes. mayactl is not used/required while provisioning or managing the OpenEBS volumes, but it is currently used while debugging and troubleshooting.\n- mayactl is the client like kubectl which requests to maya-apiserver to get specific information whereas kubectl requests to Kubernetes apiserver to get specific information.\n- mayactl helps retrieve storage related information for debugging/troubleshooting storage related issues. mayactl provides various commands to create volume, get volume details and create, list and revert snapshot and many more.\n\nTo know more about the mayactl visit:\n\n[https://openebs.io/docs/deprecated/mayactl](/docs/deprecated/mayactl)\n\n## OpenEBS Architecture\n\n![OpenEBS Architecture](/images/blog/openebs-architecture.png)\n\nTo know about the OpenEBS visit: [https://openebs.io/docs](/docs)\n\n### These few things are required to be installed in your system (with Ubuntu host) to run mayactl\n\n1. **Docker**\n\n- To install docker run these commands\n\n ```\n sudo apt-get update\n sudo apt-get install apt-transport-https ca-certificates curl software-properties-common\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository \\\n \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable\"\n sudo apt-get update\n sudo apt-get install docker.io\n ```\n\n- Or you can visit the official docker website to install docker\n\n[https://openebs.io/docs/deprecated/mayactl](/docs/deprecated/mayactl)\n\n2. **open-iscsi package**\n\n- To install open-iscsi package run these commands\n\n ```\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n ```\n\n3. **Golang**\n\n- To install golang, visit the official golang website: [https://golang.org/doc/install](https://golang.org/doc/install)\n\n4. **minikube**.\n\n- To install minikube, run these commands:\n\n ```\n # minikube requires virtualbox to be installed as a dependency\n\n sudo apt-get install virtualbox virtualbox-ext-pack\n sudo apt-get update\n\n # minikube version 0.24.0\n\n curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/\n ```\n\n5. **Kubectl**.\n\n- Run these commands to install Kubectl:\n\n ```\n curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/linux/amd64/kubectl\n chmod +x ./kubectl\n sudo mv ./kubectl /usr/local/bin/kubectl\n ```\n\n### How do I run mayactl in a local machine for development purposes?\n\n1. Open the openebs repo ( [https://github.com/openebs/openebs](https://github.com/openebs/openebs)) and star the openebs repo.. 😄 (Not Mandatory)\n\n2. Fork the openebs/openebs and openebs/maya repositories into your GitHub account.\n\n- Visit and click on the fork option (both repositories)\n\n[a) openebs/openebs](https://github.com/openebs/openebs)\n\n[OpenEBS is containerized block storage written in Go for cloud native and other environments w/ per container (or pod)…](https://github.com/openebs/openebs)[github.com](https://github.com/openebs/openebs)\n\n[b) openebs/maya](https://github.com/openebs/maya)\n\n[maya — OpenEBS Maya extends Kubernetes capabilities to orchestrate CAS containers.](https://github.com/openebs/maya)[github.com](https://github.com/openebs/maya)\n\n3. Clone the openebs and maya repositories inside your gopath. Then run these commands to clone:\n\n```\n# if directories not present create the directories in same hierarchy\n\ncd $GOPATH/src/github.com/openebs\ngit clone https://github.com//openebs.git\ngit clone https://github.com//maya.git\n```\n\n4. Run the single node cluster using the minikube command.\n\n```\nminikube start --vm-driver=none\n\n# To check whether minikube is configured and running\n\nminikube status\n```\n\n5. Install OpenEBS by executing these commands:\n\n```\ncd $GOPATH/src/github.com/openebs/openebs/k8s/\nkubectl apply -f openebs-operator.yaml\nkubectl apply -f openebs-storageclasses.yaml\n```\n\n![kubernetes commands](/images/blog/install-openebs-by-commands.png)\n\n6. Now we have the openebs-provisioner and maya-apiserver running as a pod in the Kubernetes (minikube) cluster.\n\n- To get the pods, run this command:\n\n ```\n kubectl get pods\n NAME READY STATUS RESTARTS AGE\n maya-apiserver-7b8d5496cc-kgmnn 1/1 Running 0 3m\n openebs-provisioner-6797d44769-phnnc 1/1 Running 2 3m\n ```\n\n7. To run/access mayactl, you will need to login/execute into the maya-apiserver pod on Kubernetes.\n\n- Find out the name of the maya api-server pod by running the following commands:\n\n ```\n kubectl get pods\n\n # It will access the bash shell inside the pod\n\n kubectl exec -it /bin/bash\n ```\n\n8. Now you can run all mayactl commands as you are inside the maya-apiserver pod.\n\n- Try running these commands after exececute/login into the pod.\n\n ```\n mayactl -help\n ```\n\nGo through the issues ([https://github.com/openebs/maya/issues](https://github.com/openebs/maya/issues)) and start modifying the mayactl code, located in $GOPATH/src/github.com/openebs/maya/cmd/mayactl, and start contributing to OpenEBS. Also, you can start contributing by writing a small unit test code in mayactl. For every PR you raise, you will also receive goodies from the OpenEBS team. 😃\n\n**How do I test the changes made in mayactl?**\n\n1. After modifying the mayactl code, go into the maya directory, i.e $GOPATH/src/github.com/openebs/maya, and run these commands:\n\n```\n# run this if not currently in maya directory\n\ncd $GOPATH/src/github.com/openebs/maya\n\n# this will create the mayactl binary into the bin folder inside maya directory\n\nmake mayactl\n```\n\n2. After the build has been completed, copy the mayactl binary from the bin folder to the maya-apiserver pod using the command:\n\n```\nkubectl cp $GOPATH/src/github.com/openebs/maya/bin/maya/mayactl :/tmp/\n```\n\n3. Login/execute into the maya-apiserver pod to run the mayactl binary.\n\n```\nkubectl exec -it /bin/bash\ncd /tmp/\n```\n\n4. Here the mayactl binary you copied is shown.\n\n- To run that binary, use the following command:\n\n ```\n ./mayactl -help\n ```\n\nNow you can easily see the changes you made in the mayactl command line tool. You are also now ready to raise the PR’s. 😃\n\n**Reference:**\n\n- [https://www.youtube.com/watch?v=yzMEYT-yzRU](https://www.youtube.com/watch?v=yzMEYT-yzRU)\n- [https://openebs.io/docs](/docs?__hstc=216392137.2d2e61b1b8f85b3675bfaef604437f8a.1580205967521.1580205967521.1580205967521.1&__hssc=216392137.1.1580205967522&__hsfp=2262740235)\n","notHasFeatureImage":false,"slug":"how-to-start-contributing-to-mayactl"},{"id":82,"title":"Provisioning Google Cloud with k8s using it’s in-house tool, KOPS","author":"Harshvardhan Karn","author_info":"Harshvardhan Karn works at MayaData Inc. He is a public speaker, has talked in few local meetups and at two major conferences. In his free time, he likes to play Guitar, Netflix.","date":"14-08-2018","tags":["Docker","Kubernetes","OpenEBS","Kops"],"excerpt":"Setting up and using a cluster in GCP offers a few significant advantages over using GKE. For instance, using GCP gives the user the liberty to use their custom binaries or a pure Open Source Kubernetes.","content":"\nI am very excited for this post, as I have been working on this content for a few weeks. I might sound naive throughout this, mainly because I am, to be honest. Setting up and using a cluster in _GCP_ offers a few significant advantages over using _GKE_. For instance, using GCP gives the user the liberty to use their _custom binaries_ or a _pure Open Source Kubernetes_. Also, _GKE_ does not support modification or access to the master node, whereas a manually setup-ed k8s cluster over _VMs_ does.\n\nTo gain the full understanding of Kubernetes, some people like to get their hands dirty for a course of interval. Luckily, we have a few more tools to get a Kubernetes cluster up and running in VM instances of **Google Cloud**. I find that tools like _kubernetes-incubator/kubespray , crosscloudci/cross-cloud_ and kops, are not very straightforward to use, but _kops, kubespray_ is somewhat close. _Cross-cloud_ on the other hand, has poor documentation and is not very stable. _kops_ here stands for ‘_Kubernetes Operations._’ To be honest, I find this to be a good tool to deploy the Cluster over Google Cloud Platform (**GCP**) or Amazon Web Services (**AWS**). I certainly would not say it is the best, but this tool is documented to the extent that one can use it. The original idea of KOPS was to create user a production ready cluster in **AWS**. Allowing it to provision the GCP is luxury since GCP already provides with GKE and AWS does not.\n\n### **Requirements**\n\nThese items are required to deploy the production-ready k8s cluster in GCP:\n\n- **KOPS**\n\n ```\n wget -O kops\n https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '\"' -f 4)/kops-linux-amd64\n\n chmod +x ./kops\n\n sudo mv ./kops /usr/local/bin/\n ```\n\n- **GCloud**\n\n [https://cloud.google.com/sdk/](https://cloud.google.com/sdk/)\n\n Once you are done installing GCloud SDK, you must run _gcloud init_. This will configure your gcloud with your existing GCP project.\n\n- **kubectl**\n\n From the [Official Kubernetes kubectl release:](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n\n ```\n wget -O kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl\n chmod +x ./kubectl\n sudo mv ./kubectl /usr/local/bin/kubectl\n ```\n\n- A little patience …\n\n## Let’s Begin!\n\nA quick note: every time you create a cluster, it also creates a _Virtual Private Cloud_ (**VPC**), per se. Google Cloud allows you to create only a maximum of 5 VPC’s in one project, and a total of only 5 clusters. So, to resolve this problem, we can create a **VPC** explicitly and use it as a common Network for the rest of the clusters.\n\n### Create a VPC\n\nHere I am using subnet-mode as auto, and it will create a VPC _openebs-e2e_ with a subnet in every zone.\n\n```\ngcloud compute networks create openebs-e2e --project=openebs-ci --subnet-mode=auto\n```\n\n### Create a Bucket\n\nKops needs a State Store to hold the configuration of our cluster. In our case, it is Google Cloud Storage Buckets. So, let’s create one empty Bucket using the following:\n\n```\ngsutil mb gs://openebs-dev/\n```\n\nNow, since we are ready with the Bucket, we can populate it with our cluster’s State Store, i.e. Cluster object and InstanceGroup object.\n\n**Create the Cluster & InstanceGroup Objects in Our State Store**\n\n_kops create cluster_, creates the Cluster object and InstanceGroup object. Here, we’ll be working within kops.\n\n```\nPROJECT=`gcloud config get-value project`\nexport KOPS_FEATURE_FLAGS=AlphaAllowGCE # to unlock the GCE features\nkops create cluster openebs-dev.k8s.local --zones us-central1-a\n--state gs://openebs-dev/ --project=${PROJECT}\n--kubernetes-version=1.11.1 --node-count 3\n```\n\nNow we can list the Cluster objects in our kops State Store (the GCS bucket we created):\n\n```\nkops get cluster --state gs://openebs-dev/\nNAME CLOUD ZONES\nopenebs-dev.k8s.local gce us-central1-a\n```\n\n**NB:** It is not necessary to use the same name for the Bucket and Cluster; you are free to use whatever name you wish.\n\n### Create a Cluster\n\nWe are now ready with all of the changes and the cluster configuration, so we will proceed with the creation of the cluster. _kops create cluster_ created the Cluster object and the InstanceGroup object in our State Store, but it did not actually create any instances or other cloud objects in GCE. To do that, we’ll use _kops update cluster_.\n\n_kops update cluster_ without _--yes_ will show us a preview of changes that will be made. It comes handy in case we want to see or verify the specs before creation.\n\n```\nkops update cluster openebs-dev.k8s.local --state gs://openebs-dev/ --yes\n```\n\nCheers!\n\nWe have now deployed the Kubernetes cluster on GCP. If you go to the _Compute Engine_ in _Google Cloud Platform_, you will find 4 new nodes, where 1 is the master and the rest are worker nodes. Just to save your day, if you are wondering why you could not find your nascent cluster inside _Google Kubernetes Engine_, this is not a mistake or error because it is not a GKE Cluster. All GzCP knows is that there are 4 VMs running in the project, which we know is a K8s cluster.\n\n## Out-of-the-Box\n\nWe are now ready with the cluster, but is it ready for the deployments? Once the kops is finished creating the cluster, we can validate its readiness using the following:\n\n```\nkops validate cluster --state gs://openebs-dev/\n\n\nI0808 12:34:10.238009 25907 gce_cloud.go:273] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b]\nINSTANCE GROUPS\nNAME ROLE MACHINETYPE MIN MAX SUBNETS\nmaster-us-central1-a Master n1-standard-1 1 1 us-central1\nnodes Node n1-standard-2 3 3 us-central1\nNODE STATUS\nNAME ROLE READY\nmaster-us-central1-a-067f master True\nnodes-6rt6 node True\nnodes-lvs5 node True\nnodes-wbb8 node True\nYour cluster openebs-dev.k8s.local is ready\n```\n\nIf you find that the cluster not ready, wait for a few minutes as it takes some time to configure the cluster. You can even check using _kubectl_ from your control machine:\n\n```\nkubectl get nodes\n```\n\nYou will see the node counts once your Cluster is up, viz. _kubelets_ are configured. If you are wondering how you got your _kubectl_ configured to this cluster, _kops_ does that for you. It exports a kubecfg file for a cluster from the state store to your _~/.kube/config_ local machine where you are running _kops_. If you want to export this config to some other path, you can the following:\n\n```\nkops export kubecfg openebs-dev.k8s.local\n```\n\nI wrote an Ansible playbook for [Litmus](https://github.com/openebs/litmus/), which is actually a wrapper for all of these to bring up the cluster on GCP. You can check it out here:\n\n[https://github.com/openebs/litmus/tree/master/k8s/gcp/k8s-installer](https://github.com/openebs/litmus/tree/master/k8s/gcp/k8s-installer)\n\nThe playbook also checks the cluster availability implicitly using a python script. This will hold the playbook from termination until the cluster is ready to use. _kops validate_ works well, but **not** for **k8s version < 1.9,** up to the day of writing this post.\n\nGodspeed!\n","notHasFeatureImage":false,"slug":"provisioning-google-cloud-with-k8s-using-it’s-in-house-tool,-kops"},{"id":83,"title":"Setting up WordPress and SQL with OpenEBS","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"14-08-2018","tags":["OpenEBS","Kubernetes","Cloud Native Storage","Open Source","State Department"],"excerpt":"Wordpress is a well-known blogging platform. New bloggers are often surprised when they find out how easy it is to get set up and start their first piece in this popular tool.","content":"\n[Wordpress](https://en.wikipedia.org/wiki/WordPress) is a well-known blogging platform. New bloggers are often surprised when they find out how easy it is to get set up and start their first piece in this popular tool. In this blog, we will show how to deploy WordPress and MySQL on OpenEBS in their Kubernetes cluster.\n\n## What is OpenEBS?\n\nOpenEBS offers containerized persistent block storage using Docker containers. Those blocks are often referred to as Virtual Storage Machines (similar to K8s pods). OpenEBS seamlessly provides scalable storage volumes and manages them effortlessly. For more information, you can visit [https://openebs.io/join-our-slack-community](https://openebs.io/join-our-slack-community?__hstc=216392137.b7acacf689e0cc4579eea008f86d0c72.1579857743065.1579857743065.1579857743065.1&__hssc=216392137.1.1579857743066&__hsfp=3765904294).\n\n### Prerequisites:\n\n- A k8s cluster with at least one minion.\n- Basic knowledge of writing services, deployment in k8s.\n- Kubectl, already configured.\n- A code editor for writing a yamls.\n- Brains.\n\nLet’s get started!\n\n## Setting up OpenEBS\n\nBefore starting with WordPress, we need to set up OpenEBS. For this article, I will be using OpenEBS v0.6 (you are free to use newer versions if you wish).\n\nWhen setting up OpenEBS, you need to apply the following yamls:\n\n```\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml\n```\n\nThe first yaml is for the openebs-operator, and the second one is for openebs-storage-classes. For more information look [here](/docs?__hstc=216392137.b7acacf689e0cc4579eea008f86d0c72.1579857743065.1579857743065.1579857743065.1&__hssc=216392137.1.1579857743066&__hsfp=3765904294). After applying the above yamls, the output of kubectl get pods — all-namespaces will look like this:\n\n```\n$ kubectl get pods --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\nkube-system event-exporter-v0.2.1-5f5b89fcc8-bhv7r 2/2 Running 0 16m\nkube-system fluentd-gcp-scaler-7c5db745fc-tb2zw 1/1 Running 0 16m\nkube-system fluentd-gcp-v3.0.0-wqgzx 2/2 Running 0 14m\nkube-system heapster-v1.5.3-77c6fcd568-q8txc 3/3 Running 0 15m\nkube-system kube-dns-788979dc8f-4lgsf 4/4 Running 0 16m\nkube-system kube-dns-autoscaler-79b4b844b9-jldbr 1/1 Running 0 16m\nkube-system kube-proxy-gke-ashish-ranjan-default-pool-b3a38b91-cv5d 1/1 Running 0 16m\nkube-system l7-default-backend-5d5b9874d5-hvrgb 1/1 Running 0 16m\nkube-system metrics-server-v0.2.1-7486f5bd67-hls82 2/2 Running 0 15m\nopenebs maya-apiserver-68c98fdb76-vbslv 1/1 Running 0 1m\nopenebs openebs-provisioner-5569654c96-hmhb5 1/1 Running 0 1m\nopenebs openebs-snapshot-operator-5f7c4d9bd8-7fnfv 2/2 Running 0 1m\n```\n\nWait until all openebs namespaced pods move into a running state. Once this is completed, we’ll start by creating a secret for sql.\n\n## Creating a Secret\n\n```\nkubectl create secret generic mysql-pass --from-literal=password=w0rdPres5\n```\n\nRun the above kubectl command to create a mysql password.\n\n## Wordpress Deployment\n\nNow, let's start writing the WordPress deployment yaml. Copy and save the above into a Wordpress.yaml file and execute a kubectl apply on it. Once this is done, the output of kubectl get pods,svc,pvc — all-namespaces will look similar to this:\n\n```\n$ kubectl get pods,svc,pvc --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-766678794-jtltg 2/2 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-2pkt4 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-htdxh 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-rdtlk 1/1 Running 0 2m\ndefault pod/wordpress-7bdfd5557c-5b4nh 1/1 Running 2 2m\nkube-system pod/event-exporter-v0.2.1-5f5b89fcc8-wprbm 2/2 Running 0 8m\nkube-system pod/fluentd-gcp-scaler-7c5db745fc-s9mp9 1/1 Running 0 8m\nkube-system pod/fluentd-gcp-v3.0.0-49vq6 2/2 Running 0 5m\nkube-system pod/fluentd-gcp-v3.0.0-kfjsx 2/2 Running 0 5m\nkube-system pod/fluentd-gcp-v3.0.0-tg5hh 2/2 Running 0 5m\nkube-system pod/heapster-v1.5.3-76f7f5f544-z7xk9 3/3 Running 0 6m\nkube-system pod/kube-dns-788979dc8f-2chf4 4/4 Running 0 8m\nkube-system pod/kube-dns-788979dc8f-ls4ln 4/4 Running 0 7m\nkube-system pod/kube-dns-autoscaler-79b4b844b9-8745z 1/1 Running 0 8m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8866 1/1 Running 0 7m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8fh5 1/1 Running 0 7m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-9jjf 1/1 Running 0 8m\nkube-system pod/l7-default-backend-5d5b9874d5-k9mn8 1/1 Running 0 8m\nkube-system pod/metrics-server-v0.2.1-7486f5bd67-wlnlg 2/2 Running 0 6m\nopenebs pod/maya-apiserver-68c98fdb76-5bt2z 1/1 Running 0 4m\nopenebs pod/openebs-provisioner-5569654c96-4n4hp 1/1 Running 0 4m\nopenebs pod/openebs-snapshot-operator-5f7c4d9bd8-shk58 2/2 Running 0 4m\nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndefault service/kubernetes ClusterIP 10.55.240.1 443/TCP 8m\ndefault service/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.253.18 3260/TCP,9501/TCP 2m\ndefault service/wordpress LoadBalancer 10.55.253.170 104.154.224.12 80:31392/TCP 2m\nkube-system service/default-http-backend NodePort 10.55.243.156 80:30045/TCP 8m\nkube-system service/heapster ClusterIP 10.55.248.200 80/TCP 8m\nkube-system service/kube-dns ClusterIP 10.55.240.10 53/UDP,53/TCP 8m\nkube-system service/metrics-server ClusterIP 10.55.244.66 443/TCP 8m\nopenebs service/maya-apiserver-service ClusterIP 10.55.240.121 5656/TCP 4m\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\ndefault persistentvolumeclaim/wp-pv-claim Bound pvc-7030ae5f-9d40-11e8-afcb-42010a800179 20Gi RWO openebs-standard 2m\n```\n\n## MySql deployment\n\nFollow the same procedure as done for WordPress deployment and execute a kubectl apply on it. The output of kubectl get pods,svc ,pvc — all-namespaces will look similar to this:\n\n```\n$ kubectl get pods,svc,pvc --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-ctrl-b5c4f588f-lrl2l 2/2 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-2jcgx 1/1 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-j5tsk 1/1 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-q9kww 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-766678794-jtltg 2/2 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-2pkt4 1/1 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-htdxh 1/1 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-rdtlk 1/1 Running 0 6m\ndefault pod/wordpress-7bdfd5557c-5b4nh 1/1 Running 4 6m\ndefault pod/wordpress-mysql-bcc89f687-zlt5q 1/1 Running 0 2m\nkube-system pod/event-exporter-v0.2.1-5f5b89fcc8-wprbm 2/2 Running 0 11m\nkube-system pod/fluentd-gcp-scaler-7c5db745fc-s9mp9 1/1 Running 0 11m\nkube-system pod/fluentd-gcp-v3.0.0-49vq6 2/2 Running 0 9m\nkube-system pod/fluentd-gcp-v3.0.0-kfjsx 2/2 Running 0 9m\nkube-system pod/fluentd-gcp-v3.0.0-tg5hh 2/2 Running 0 9m\nkube-system pod/heapster-v1.5.3-76f7f5f544-z7xk9 3/3 Running 0 10m\nkube-system pod/kube-dns-788979dc8f-2chf4 4/4 Running 0 11m\nkube-system pod/kube-dns-788979dc8f-ls4ln 4/4 Running 0 10m\nkube-system pod/kube-dns-autoscaler-79b4b844b9-8745z 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8866 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8fh5 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-9jjf 1/1 Running 0 11m\nkube-system pod/l7-default-backend-5d5b9874d5-k9mn8 1/1 Running 0 11m\nkube-system pod/metrics-server-v0.2.1-7486f5bd67-wlnlg 2/2 Running 0 10m\nopenebs pod/maya-apiserver-68c98fdb76-5bt2z 1/1 Running 0 7m\nopenebs pod/openebs-provisioner-5569654c96-4n4hp 1/1 Running 0 7m\nopenebs pod/openebs-snapshot-operator-5f7c4d9bd8-shk58 2/2 Running 0 7m\nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndefault service/kubernetes ClusterIP 10.55.240.1 443/TCP 12m\ndefault service/pvc-082a54c8-9d41-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.251.84 3260/TCP,9501/TCP 2m\ndefault service/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.253.18 3260/TCP,9501/TCP 6m\ndefault service/wordpress LoadBalancer 10.55.253.170 104.154.224.12 80:31392/TCP 6m\ndefault service/wordpress-mysql ClusterIP None 3306/TCP 2m\nkube-system service/default-http-backend NodePort 10.55.243.156 80:30045/TCP 11m\nkube-system service/heapster ClusterIP 10.55.248.200 80/TCP 11m\nkube-system service/kube-dns ClusterIP 10.55.240.10 53/UDP,53/TCP 11m\nkube-system service/metrics-server ClusterIP 10.55.244.66 443/TCP 11m\nopenebs service/maya-apiserver-service ClusterIP 10.55.240.121 5656/TCP 7m\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\ndefault persistentvolumeclaim/mysql-pv-claim Bound pvc-082a54c8-9d41-11e8-afcb-42010a800179 20Gi RWO openebs-standard 2m\ndefault persistentvolumeclaim/wp-pv-claim Bound pvc-7030ae5f-9d40-11e8-afcb-42010a800179 20Gi RWO openebs-standard 6m\n```\n\nIf all of the pods are running, check the external IP of the WordPress load balancer. For example, in my case it is **104.154.224.12**. Open the your web browser IP if you are redirected to your WordPress setup page. Congratulations! Your WordPress is now ready for you to start your blog!\n\nHappy blogging!\n\n### Proof:\n\n![wordpress](/images/blog/wordpress.png)\n","notHasFeatureImage":false,"slug":"setting-up-wordpress-and-sql-with-openebs"},{"id":84,"title":"ARMing Kubernetes with OpenEBS \\#1","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"02-08-2018","tags":["Arm64","Kubernetes","Lepotato","Owncloud","Solutions"],"excerpt":"Running stateful containers on Le Potato. Why not! It’s fun and extremely efficient! I know many people (including me) used to run ownCloud on their desktop computers.","content":"\n## Running stateful containers on Le Potato\n\nWhy not! It’s fun and extremely efficient! I know many people (including me) used to run ownCloud on their desktop computers. I have finally decided to retire my old desktop computer and was looking ways to keep my ownCloud instance alive and maybe even improve it a bit.\n\nFirst, I ran **ownCloud** on GKE Kubernetes cluster and came to a decison quickly that it’s not what I needed:\n\n- I am used to the speed of USB 3.0 for large uploads when needed. I wanted to keep the option of (50MB+/sec) using USB. Which means, if I choose a budget ARM SoC route, then the board should have **non-shared bandwidth** for LAN and USB.\n- 4 node GKE cluster using n1-standard-1 + 200GB storage costs ~$225/month, I would rather use Dropbox for $20/month = 240$/year (still doesn’t give me what I need).\n- **Low-power**, possibly solar-powered. I’ll share my power consumption findings in the series of blog articles.\n- Everything on **Kubernetes** is more fun, right?\n\nI was looking into [Raspberry Pi 3 Model B](http://amzn.to/2GMjYt4) option and after a quick trial realized that shared USB/Ethernet bandwidth and lack of MicroSD UHS support is not going to give me the performance I need and found the **AML-S905X-CC Le Potato** board.\n\nLibre Computer Board, code name Le Potato, is designed as a drop in hardware replacement for the Raspberry Pi 3 Model B (In fact it has the exact same form factor and port locations) and offers faster performance, more memory, lower power, higher IO throughput, 4K capabilities, open market components, improved media acceleration and removal of the vendor locked-in interfaces. This platform uses the latest technologies and is built upon proven long-term available chips. It is supported by upstream Linux and has a downstream development package based on Linux 4.9 LTS that offers ready-to-go 4K media decoding, 3D acceleration, and more.\n\nMost importantly, Le Potato has almost double the performance of RPi 3 with 2GB memory and 50% faster CPU and GPU. It also has non-shared bandwidth for LAN and USB and MicroSD UHS support. I was able to get over 70MB/s read&write performance vs ~15–20MB/s on RPi 3. I also noticed that even under heavy load Le Potato has lower power consumption compared to Rpi 3.\n\nIt sounds too good to be true, right? Since Le Potato is new, I’ve decided to run both side-to-side and publish my experience.\n\nIn this blog post, I will focus on setting up a Kubernetes on a Le Potato Clusters, install ownCloud on top, and compare it to Rpi 3.\n\n## Prerequisites\n\n### Hardware\n\n- 4 x [Libre Computer Board AML-S905X-CC (Le Potato) 64-bit (2GB)](http://amzn.to/2ptxGJS) $45\n- 4 x 32GB MicroSD Card ([Samsung 32GB 95MB/s MicroSD](http://amzn.to/2uayTe4) $9.99)\n- 4 x 128GB USB Drive ([Samsung 128GB USB 3.0 Flash Drive Fit (MUF-128BB/AM)](http://amzn.to/2psgFPC) $39)\n- 1 x Desktop Switch ([TP-link 5-Port Gigabit Desktop Switch](http://amzn.to/2u3TCQN) $29.99)\n- 1 x Active USB Hub ([Generic 7-Port USB Hub with ON/OFF Switch](http://amzn.to/2IBIZaO) $5.64)\n- 4 x short USB to Micro USB cable ([ZiBay Micro USB Short Sync Cable for Select Models/Device, 7-Inch — Pack of 5](http://amzn.to/2G8Doub) $6.99)\n- For comparison: 4 x [Raspberry Pi 3 Model B](http://amzn.to/2GMjYt4) $34.62\n\n### Optional:\n\n- Short cables make it look clean and nice:\n 6-inch CAT6 flat network cables ([5-PACK 6-inch CAT6 Network UTP Ethernet RJ45 Flat-Design](http://amzn.to/2GcT5AV) $12.48)\n- One touchscreen to access the cluster when nothing else available:\n 1x 3.5 inch TFT Touch Screen ([kuman 3.5 Inch 480×320 TFT Touch Screen Monitor for Raspberry Pi](http://amzn.to/2pwR9tt) $19.39)\n- I also build a mobile version to run in my car using this with a Tmobile line ([SIM800 Module GSM GPRS Expansion Board UART V2.0](http://amzn.to/2GLBeyE) $25.99)\n\n### Software components used\n\n- [Armbian 5.38 Ubuntu Xenial](https://dl.armbian.com/lepotato/) (for Le Potato)\n- [Raspbian Stretch Lite](https://www.raspberrypi.org/downloads/raspbian/) 2017–11–29 (for Rpi 3)\n- [Etcher](https://etcher.io/) v1.3.1\n- Kubernetes v1.9.2+\n- OpenEBS 0.5.3 arm64\n- ownCloud\n\nI will start with Le Potato and compare against Raspberry Pi 3 on my next blog.\n\n### Flash Le Potato Armbian image on SD Cards\n\n**Armbian** provides Debian and Ubuntu based builds for ARM development boards. Armbian Ubuntu is pretty much same as Ubuntu, except the desktop interface. Armbian uses the Xfce desktop environment, which is a lighter than Gnome or KDE. Its main advantage is its speed, and it’s ideal for systems with 256 MB to 512 MB of RAM. And, I plan to disable desktop anyways.\n\nDownload the [Armbian Ubuntu image](https://dl.armbian.com/lepotato/Ubuntu_xenial_next_desktop.7z) from the link [here](https://dl.armbian.com/lepotato/), extract and burn it to your SD cards using Etcher.\n\n![Etcher flashing](https://cdn-images-1.medium.com/max/800/0*XI8xSg4dCl_IWvbz.png)\n\nPlug the SD card into your Le Potato board and power on.\n\n![Terminal Window](https://cdn-images-1.medium.com/max/800/0*ojAbBScZY7giAV7b.jpg)\n\nLogin as `root` and use password `1234`. You will be prompted to change this password at first login. Next, you will be asked to create a normal user account that is sudo enabled.\n\n### Prepare Armbian host\n\nAfter reboot, you will auto login to your host with the new user you have created.\n\n![Armbian Le Potato](https://cdn-images-1.medium.com/max/800/0*UOLgX8I0Oz9hw7I8.jpg)\n\nChange the hostname and set static IP by using armbian-config utility:\n\n`sudo armbian-config`\n\n![armbian-config utility](https://cdn-images-1.medium.com/max/800/0*s8fjc1-L-9pkDzRE.png)\n\nDisable swap by running the following commands:\n\n```\nsudo systemctl disable zram-configsudo swapoff -a\n```\n\nAnd also comment out the reference to swap in /etc/fstab file:\n\n```\nsudo vi /etc/fstab\n```\n\nAfter reboot, confirm that swap space is disabled by running the following command. It should return empty.\n\n```\nsudo swapon — summary\n```\n\nInstall Golang 1.10:\n\n```\nwget https://dl.google.com/go/go1.10.linux-arm64.tar.gz\nsudo tar -C /usr/local -xzf go1.10.linux-arm64.tar.gz\nexport PATH=$PATH:/usr/local/go/bin\nmkdir go\nexport GOPATH=”$HOME/go”\ngo get github.com/kubernetes-incubator/cri-tools/cmd/crictl\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Install Docker on Armbian Ubuntu (arm64)\n\nRun the following command to install Docker on all nodes. The second line is to use Docker as a non-root user, use your username instead of mine below (murat):\n\n```\ncurl -sL https://get.docker.com | sh\nsudo usermod murat -aG docker\n```\n\nSuccessful installation would look like below:\n\n```\nmurat@kubenode1:~$ curl -sL https://get.docker.com | sh\n # Executing docker install script, commit: 02d7c3c\n + sudo -E sh -c apt-get update -qq >/dev/null\n + sudo -E sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null\n + sudo -E sh -c curl -fsSL “https://download.docker.com/linux/ubuntu/gpg\" | apt-key add -qq →/dev/null\n + sudo -E sh -c echo “deb [arch=arm64] https://download.docker.com/linux/ubuntu xenial edge” > /etc/apt/sources.list.d/docker.list\n + [ ubuntu = debian ]\n + sudo -E sh -c apt-get update -qq >/dev/null\n + sudo -E sh -c apt-get install -y -qq — no-install-recommends docker-ce >/dev/null\n + sudo -E sh -c docker version\nClient:\n Version: 18.02.0-ce\n API version: 1.36\n Go version: go1.9.3\n Git commit: fc4de44\n Built: Wed Feb 7 21:11:48 2018\n OS/Arch: linux/arm64\n Experimental: false\n Orchestrator: swarm\nServer:\n Engine:\n Version: 18.02.0-ce\n API version: 1.36 (minimum version 1.12)\n Go version: go1.9.3\n Git commit: fc4de44\n Built: Wed Feb 7 21:09:57 2018\n OS/Arch: linux/arm64\n Experimental: false\nIf you would like to use Docker as a non-root user, you should now consider\nadding your user to the “docker” group with something like:\nsudo usermod -aG docker murat\nRemember that you will have to log out and back in for this to take effect!\nWARNING: Adding a user to the “docker” group will grant the ability to run\n containers which can be used to obtain root privileges on the\n docker host.\n Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface\n for more information.\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Install Kubernetes on Armbian for Le Potato\n\nRun the following command to install Kubeadm on all nodes:\n\n```\ncurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add — && \\\n echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee /etc/apt/sources.list.d/ kubernetes.list && \\\n sudo apt-get update -q && \\\n sudo apt-get install -qy kubeadm\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Initialize Kubernetes master node\n\nInitialize your master K8s node:\n\n```\nsudo kubeadm init — pod-network-cidr=10.20.0.0/24 — apiserver-advertise-address=10.10.0.131\n```\n\nBy default, token expires in 24h. If you need it longer, then you can add `— token-ttl=0` to the end of the command above to generate token that does not expire.\n\nThis step may take around 10 minutes and after that, you will see a summary like below:\n\n```\n…\nYour Kubernetes master has initialized successfully!\nTo start using your cluster, you need to run the following as a regular user:\nmkdir -p $HOME/.kube\n sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n sudo chown $(id -u):$(id -g) $HOME/.kube/config\nYou should now deploy a pod network to the cluster.\n Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:\n https://kubernetes.io/docs/concepts/cluster-administration/addons/\nYou can now join any number of machines by running the following on each node\nas root:\nkubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n```\n\nRun the following:\n\n```\nmkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\nDeploy a pod network to the cluster. I used flannel, you can see your other options [here](https://kubernetes.io/docs/concepts/cluster-administration/addons/).\n\n```\nsudo sysctl net.bridge.bridge-nf-call-iptables=1\nkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml\n```\n\nBy default, pods cannot be scheuled on the master node. If you want to be able to schedule pods on the master, e.g. for a single-machine Kubernetes cluster for development, run:\n\n```\nkubectl taint nodes — all node-role.kubernetes.io/master-\n```\n\nAs soon as the pod network has been installed, you can continue by joining your nodes.\n\nTo confirm that kube-dns pod is up run the command below and check the output:\n\n```\nmurat@kubenode1:~$ kubectl get pods — all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n kube-system etcd-kubenode1 1/1 Running 0 1m\n kube-system kube-apiserver-kubenode1 1/1 Running 0 1m\n kube-system kube-controller-manager-kubenode1 1/1 Running 0 1m\n kube-system kube-dns-6448b967fc-bc58z 3/3 Running 0 1m\n kube-system kube-proxy-h7p6s 1/1 Running 0 1m\n kube-system kube-scheduler-kubenode1 1/1 Running 0 1m\n [/cce_bash]\n```\n\nNote: If kube-dns is stuck in the Pending state. Follow the steps below to fix it and re init your master. This issue and the solution was mentioned [here](https://github.com/kubernetes/kubernetes/issues/43815).\n\n```\nkubeadm reset\nsudo nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n```\n\nRemove the `$KUBELET_NETWORK_ARGS` entry from the ExecStart, save the file, and reload systemd and kube services.\n\n```\nsystemctl daemon-reload\nsystemctl restart kubelet.service\n```\n\nInitialize your master K8s node again.\n\n## Join Kubernetes nodes to the cluster\n\nYou can now join any number of nodes by running the command with the token generated during the K8s master initialization:\n\n```\nmurat@kubenode2:~$ kubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n [preflight] Running pre-flight checks.\n [preflight] Some fatal errors occurred:\n [ERROR IsPrivilegedUser]: user is not running as root\n [preflight] If you know what you are doing, you can make a check non-fatal with ` — ignore-preflight-errors=…`\n murat@kubenode2:~$ sudo kubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n [preflight] Running pre-flight checks.\n [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.0-ce. Max validated version: 17.03\n [discovery] Trying to connect to API Server “10.10.0.131:6443”\n [discovery] Created cluster-info discovery client, requesting info from “https://10.10.0.131:6443\"\n [discovery] Requesting info from “https://10.10.0.131:6443\" again to validate TLS against the pinned public key\n [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server “10.10.0.131:6443”\n [discovery] Successfully established connection with API Server “10.10.0.131:6443”\nThis node has joined the cluster:\n * Certificate signing request was sent to master and a response\n was received.\n * The Kubelet was informed of the new secure connection details.\n```\n\nRun `kubectl get nodes` on the master to see this node join the cluster.\n\nIf you forgot the cluster token, you can generate a new one with the command:\n\n```\nkubeadm token generate\n```\n\nRepeat all the steps above on all your nodes.\n\n## Install OpenEBS on ARM (Le Potato)\n\nSimilar to most of the arm based hobby boards, Le Potato doesn’t provide any additional redundancy. Even using a RAID protected external USB device wouldn’t give me protection against node failure unless it’s some form of a shared network storage. They are both way over my affordability requirement. All I need is a replicated block device, so my container can survive a node or USB device failures.\n\nOpenEBS provides a great solution for modern x64 architecture but currently doesn’t have a build for [arm64](https://en.wikipedia.org/wiki/ARM_architecture#64/32-bit_architecture) (armv8) architecture. Therefore, I’ve opened an issue [here](https://github.com/openebs/openebs/issues/1295) and started working on it myself. I did successfully build OpenEBS images for arm64 architecture from the repo base on the 0.5.3 release and uploaded custom images to my personal docker registry [here](https://hub.docker.com/u/muratkarslioglu/). So, it is work in progress and please use it at your own risk, until it’s merged.\n\n```\nsudo apt-get install -y curl open-iscsi\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/openebs-operator-arm64.yaml\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/openebs-storageclasses.yaml\n```\n\nNow, get the list of storage classes using the below command:\n\n```\n$ kubectl get sc\n NAME PROVISIONER AGE\n openebs-cassandra openebs.io/provisioner-iscsi 1h\n openebs-es-data-sc openebs.io/provisioner-iscsi 1h\n openebs-jupyter openebs.io/provisioner-iscsi 1h\n openebs-kafka openebs.io/provisioner-iscsi 1h\n openebs-mongodb openebs.io/provisioner-iscsi 1h\n openebs-percona openebs.io/provisioner-iscsi 1h\n openebs-redis openebs.io/provisioner-iscsi 1h\n openebs-standalone openebs.io/provisioner-iscsi 1h\n openebs-standard openebs.io/provisioner-iscsi 4d\n openebs-zk openebs.io/provisioner-iscsi 1h\n```\n\n**Voila…!**\n\n`openebs-standard` storage class creates 3 replicas. That’s what I will use for my application.\n\nTo test the OpenEBS installation you can try my Jenkins example here:\n\n```\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/demo/jenkins/jenkins-arm64.yaml\n```\n\n## Next — Installing containerized OwnCloud on OpenEBS\n\nFinding right container images to run on arm64 architecture is challenging. On the next article, I will build an OwnCloud container image running on Postgres database and both containers will store their data on OpenEBS persistent volumes.\n\nMy final goal is to build a mobile OwnCloud cluster installed in my family van, where storage is replicated to another cluster in my home lab.\n\nStay tuned!\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/arming-kubernetes-with-openebs-1/)_._\n","notHasFeatureImage":false,"slug":"arming-kubernetes-with-openebs-\\#1"},{"id":85,"title":"Recap of Google Next’18","author":"Murat Karslioglu","author_info":"No author information","date":"01-08-2018","tags":["Cloud Services Platform","Knative","Google Next18","Istio","OpenEBS"],"excerpt":"This year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.","content":"\n![You know that you are at the right event when you see a familiar face like Mr. Hightower :)](https://lh3.googleusercontent.com/iBQD9nOCN5cmrzn73zLeMoHDdhbTZWa3d4sSC1k1wkudXXL0L0912hrjUe2Bxr3MBTLOM_-LDC-ZrA-zNq8arcTJfD_V6e0pc_A9_oKcm6tAsBnIfqXdTfEbnmb8Qu_PoSyBZkVN)\n\nThis year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.\n\nNext ’18 was an excellent representation of Google as a company and their culture. Sessions were mostly in Moscone West, but the whole event was spread across Moscone West, the brand new South building, and six other buildings.\n\n![Next ’18 Event Map](https://lh4.googleusercontent.com/3ojmOPqqjEieE6GxfEjgFxRRv4sIzQpA_x21hFRpj3IRrmy6i7HL4k5FO2zztbwf9b5HJlrzO8BP3bWkOM34gZQdKS5lmLqR0FjmHJr96VIToFfc-SWdIKmlLcMJLz2y_tWPbERn)\n\nThe floor plan was fun and casual; catering was of “Google Quality” and the security was insane, with metal detectors, police, K9 search dogs, and cameras everywhere. And of course games, fun, and even “Chrome Enterprise Grab n Go” were there in case you needed a loaner laptop to work on — see some pictures at the end. :)\n\n### **What I learned at the Next ’18 conference**\n\nFirst of all, a big shout out to all involved in the [**Istio** project](https://istio.io/). It is not a surprise that we see great advocate marketing and support for the Istio 1.0 GA release on social media last week. Istio is a big part of the [**Google’s Cloud Services Platform**](https://cloud.google.com/solutions/cloud-services-platform/)(**CSP**) puzzle.\n\n![GCSP Dashboard — After deploying my first app in less than 30 seconds.](https://lh4.googleusercontent.com/taoSgNELqkMCnfsqUd84nPfbIATkjucboLYdbzMUWKct5ZFiXb_PZFjoU5KFBc5LZNBz6mhuwHXEpMs49tREabCzrMDCuNTrKniQ4UYuk_i1pNxR08pUHOEjtZ3nxfUOZswcA_xe)\n\nLater this year, Google is targeting to make all components of their CSP available (in some form). CSP will combine **Kubernetes**, **GKE**, **GKE On-Prem** and **Istio** with Google’s infrastructure, security, and operations to increase velocity, reliability, and manage governance at scale.\n\nCloud Services Platform will be extensible through an open ecosystem. **Stackdriver Monitoring** and **Marketplace** are the extensions to platform services. [Marketplace](https://console.cloud.google.com/marketplace/browse?filter=solution-type:k8s) already has 27 Kubernetes apps including commonly used components of many environments such as Elasticsearch and Cassandra.\n\n![CSP Marketplace](https://lh3.googleusercontent.com/esTW1l0iBV-Wvleoosxha1W5KqmA5BQLZ-4jyfb3e0W2j_S5rzqtncJCFA8t6brQc_ZJdF2eVqaXAdhHASBlTq9izYO85SLSZRyE8mbwoB1EiFHTmQdwDHsnTdFm2EDb0i4yefVA)\n\nUsers will be able to deploy a unified architecture, that spans from their private cloud, using Google CSP to Google’s public cloud. Again, the two most important pieces to this puzzle are managed versions of the open source projects Kubernetes and Istio. To me, the rest of it still feels mostly to be DIY-like quality.\n\nKnative, Cloud Build, and CD are other significant solutions announced at Next’18.\n\n### **A new cloud availability zone, this time in your datacenter — which might be in your garage**\n\nAt first, **GKE on-prem** got me interested. But, after talking to a few Google Cloud Experts again, I felt it’s very early to be seriously considered. You can read others’ thoughts here on [Hacker News](https://news.ycombinator.com/item?id=17602555).\n\n![Discussions on GKE on-prem](https://lh5.googleusercontent.com/q_0yHazRpRZ9bLptYCw1_GQsmdM8TpbM7xXmZ8nL8nejR3uhVg79bvJokA_BrQ91VfzYYl8OLtQ1Evl5VNJqDwJM3EKwqKFsn_jr99N91hCEa1lkazjMZE3aphRbr21LEc0atqTr)\n\nGKE on-prem alpha will support **_vSphere 6.5_** only, no bare-metal for now!\n\nFailover from on-prem -> GKE is something Google team is working on. This means GKE on-prem instance will look like another availability zone (AZ) on a Google Cloud dashboard.\n\nOther than vSphere dependency, the idea of being able to have an availability zone, local in your data center is really compelling. It is also a very common use-case for [**OpenEBS**](https://openebs.io/) since there is no cloud vendor provided, a cloud-native way of spreading your cloud volumes, EBS, etc. across AZs — we see many community users running web services today using OpenEBS to enable that.\n\n### **Github and Google Partnership to provide a CI/CD platform**\n\n**Cloud Build** is Google’s fully managed CI/CD platform that lets you build and test applications in the cloud. Cloud Build is fully integrated with GitHub workflow, simplifies CI processes on top of your GitHub repositories.\n\n![Me deploying myself on Serverless Cloud Maker :)](https://lh3.googleusercontent.com/Vid2Mpm0eaSATtriAt3eLoDdBvvRcv7WCJeNBKxe_VOhVcbdrmh_nJIn5aiQlnMfEOpywRMhHF7Gnv58Nyu_5MQHoWWfxMCmPYdfDlYlKkiQPldJvHxEk9Qa5BOQBuDQNW-YZ0dc)\n\nCloud Build features;\n\n**Multiple environment support** lets developers build, test, and deploy across multiple environments such as VMs, serverless, Kubernetes, or Firebase.\n\n**Native Docker support** means that deployment to Kubernetes or GKE can be automated by just importing your Docker files.\n\n**Generous free tier**— 20 free build-minutes per day and up to 10 concurrent builds may be good enough for many small projects.\n\n**Vulnerability identification** performs built-in package vulnerability scanning for Ubuntu, Debian, and Alpine container images.\n\n**Build locally or in the cloud** enables more edge usage or GKE on-prem.\n\n### Serverless — here we are again\n\n**Knative** is a new open-source project started by engineers from Google, Pivotal, IBM, and a few others. It’s a K8s-based platform to build, deploy, and manage serverless workloads.\n\n_“The biggest concern on Knative is the dependency on Istio.”_\n\nTraffic management is critical for serverless workloads. Knative is tied to Istio and can’t take advantage of the broad ecosystem. This means existing external DNS services and cert-managers cannot be used. I believe, Knative still needs some work and not ready for prime-time. If you don’t believe me, read the installation YAML file — I mean the 17K lines “human readable” configuration file ([release.yaml](https://github.com/knative/serving/releases/download/v0.1.0/release.yaml)).\n\n![](https://lh6.googleusercontent.com/0qn1GCe8B-15DIr5G7eqqbg3FfnOcm58iQ08ZUobrKJ82xIArtNjnSuFS2KOkkEhyGfyTH8pz5_NXZOk87EllIjN4rSVYlyxxmN6iDemZ0AgM_Yd-FMZzMR-nQdCHpFPTIL84hwS)\n\n### **My take on all of the above — Clash of the Cloud Vendors**\n\nIf you have been in IT long enough, you could easily see the pattern and predict why some technologies will become more important and why will the others be replaced.\n\n_“What is happening today in the industry is the battle to become the “Top-level API” vendor.”_\n\n20–25 years ago hardware was still the king of IT. Brand-name server, network, and storage appliance vendors were ruling in the datacenters. Being able to manage network routers or configure proprietary storage appliances were the most wanted skills. We were talking to hardware…\n\n20 years ago (in 1998), VMware was founded. VMware slowly but successfully commercialized hypervisors and virtualized the IT. They became the new API to talk to, everything else under that layer became a commodity. We were suddenly writing virtualized drivers, talking software-defined storage and networking — the term “software-defined” was born. Traditional hardware vendors lost the market and momentum!\n\n12 years ago, the AWS platform was launched. Cloud vendors became the new API that developers wanted to talk to, hypervisors became a commodity. CIO and enterprises that are sucked into the cloud started worrying about the cloud lock-in. Just like the vendor lock-in or hypervisor lock-in, we have experienced before. Technology might be new, but concerns were almost the same.\n\n4 years ago, Kubernetes was announced and v1.0 released in mid-2015. Finally, an open-source project that threatens all previous, proprietary, vendor managed “Top-level API” that we were using became a majorly adopted container orchestration technology. Although it came from Google, it took off after it got open-sourced and probably would be right to say that so far financially, Red Hat profited most from Kubernetes with their Red Hat OpenShift platform. And now we see somewhat of a battle over APIs to be used in operating applications on Kubernetes, with the RedHat / CoreOS operator framework and other projects including one supported by Google and others such as Rook.io emerging to challenge or extend the framework.\n\nGoogle Container Engine (**GKE**), Microsoft Azure Container Service (AKS**)**, Amazon Elastic Container Service (**EKS**), IBM Cloud Container Service (**CCS**), and Rackspace Kubernetes-as-a-Service (**KaaS**) are all competing in the hosted Kubernetes space (new vendors expected here).\n\nThere is enough space to grow in the self-hosted Kubernetes space. GKE on-prem is the validation from Google.\n\n**Hardware>Virtualization>Cloud>Containers>Serverless???**\n\nMany of us see **Serverless** as the next step, but it might be too granular to support larger adoption and current limitations validate the claims. It doesn’t scale well for intense workloads.\n\nOne size doesn’t fit all, there are still traditional use cases that even run on bare-metal and VMs. Same might be true for Serverless. It is not for every workload. Modernizing existing workloads will take time, and we will see who will become the leader of the next “Top-level API”.\n\nWhat do you think? Who is going to win the clash of the titans? What did you think about Next’18 and Google’s strategy?\n\nThanks for reading and for any feedback.\n\n### **Some Next’18 moments from my camera**\n\n![](https://lh4.googleusercontent.com/ARfwggxkEIm1I-QXUGinQGV0zVQLzaTaQ9WxUEC4nN-xuTUsK0I-Bi4JO9kwyIi6MQYxnu0hBQDxdbkVy5nsTd5oQMEl-JCXRvdWWVhcrbCK3EfM8EegXImT2_Kn0kXeZoHbfLmK)\n\n![](https://lh5.googleusercontent.com/bUCoD0IKci0QEETmHlcrUN-wOSLFSYsIuR4aG96D3QtZo23_gm10SfBqMzUtHVduFt-XzA4m9mI-sae4ktxJRIG9m9aBUt9VUtG0ytYyjFoh-Q2GbFNlQC7Ry0iBiTiaKUNuKsFd)\n\n![](https://lh3.googleusercontent.com/Vdr19dKixjLs64xGxB_thJ3D8_-cnMihbC-gH50S5FuFJ13y2UMb42zSQy9Rp2LT9olLP5TPSRb4WPoW71l71NMJoBeK70SiFtYgsDs1l2k2tmLTiqJTlo9ajj0F0xTp4JlkQuF2)\n\n![](https://lh4.googleusercontent.com/GGkcm_nFqb_gDKJuBJISMI3mueZx6xiBE6R8diM84xEOnYmcSDQVNPaRTbIFgBf-fh1Y_8JcioxxP8g8RzrxYEUJ4Xhw51RMRdXw3aS3fVpbH_mjc_kPUC0pXL3WxXZNlgR3baCb)\n\n![](https://lh4.googleusercontent.com/seBUEu7ltrIHW3VxY_V5rqekHkLXfLYhR0dZZbsyr8MC-vww-_rGvQA3rPz0kfbqudV-LNAi292BTwNrGIe_KG56Vqr-sHSv-mPlIy8nhnJDQPQKnUj7ohg3uql67RVvqTf-earr)\n\n![](https://lh6.googleusercontent.com/2BPdWJYXZizBdmFra0GYqedeArDGynas8VsxPkC0FWhyCjUm8A7TYLpPN06hPMmQ1UQ_yPoG8mH91eRjAyQE6sbw4Jo2wjPlsqDVTLEohowMOtSQaEuSWZO0lIntDOeMip75K20P)\n\n![](https://lh3.googleusercontent.com/esI-PdVDc9sEqPoOkwG4nQhcD_FYSxs8Z1eBiHW9UTBDNO9bbd145X9vwnQgijXHTiz6DUD_bgkz9ViC1C2ukDYtjLaHVAIFlEMPcHmQEWjpKeq3pvEy4HyWXeK65oe61LUMTZ1-)\n\n![](https://lh5.googleusercontent.com/KtS0m1tRBvjWehJSCSFItgtvDK5IiAEU20aa3GfSK6TmlyPVWjQpjnq_z5OAxsa1-L7PQuNRuiK2ZRX1It-1CrDlqzv1ubwrYaZA_gQQGxsb2rXJCAiQPjZ9GLiqXHZsCess6dYz)\n\n![](https://lh6.googleusercontent.com/Uee-fb3QJewe05s4AWM2bF6b7MuRI8XgU9r3KfX72RNwVYYecjt5UX15vw1jhk0LoqvgL1MN5yKT5t9Mei6QzI7bohUAKtoQthG02YCr1VXiM4HFB-RRmQB29uuANQNDiq1sGKGe)\n\n![](https://lh6.googleusercontent.com/OXx7ZGSWDI5z2UnqunkcWtB1MWY5ZtXs3EmruVAqfZo4JiLzhd00hmRkKZi_y2Icv6FV4CJRJW68HK1laKNXCKnGI5A9Z1l53R1BtOiM6dLzjDvecuWLgzPIgir3Q89qxkHt50yo)\n\n![](https://lh4.googleusercontent.com/LUlB3APRP1t0gEEzaXmPbUxZFKGD1nVHRtdNMoKp7iSTLAZAHXOF3W_VPaZs9-XsLdw54GC2TnjnxGyR_spqek8X5ZLFkWICZpP4oXEzATj4n_vfvLQtr0FNceiT3lKzGKg7oh1f)\n\n![](https://lh4.googleusercontent.com/AZP7UsS_i0gXvSBRuoDas6D48cBdU9JD3N39C53YEIdJwXARCyEg2sXRSuvOGUSa07o0xZ2UzQc8mLgZxj6EQm8uRLZ3JXVDzEwx6kTT3Vq-eia5OV865zc0q3bq3htEDZxUiIX1)\n\n![](https://lh4.googleusercontent.com/GkXUs0HYRwS66hSMex7NDDT6Ck0NaxO4VrIHW23GzjrQbRMWDDA4EGwIOwSsg3O8tL_iNuISRywMDuF1u61rZm4fzaBisfUkyo-aPIcCcTk3KwUPunnPwgrV6-oRca0td-eYHxss)\n\n![](https://lh3.googleusercontent.com/EPOoqoV_b11WJCV9cNJFgAtGfuRyqF3xZpKKoJ46J54wNsrE0kFFp82WFZn_gRoRTSLYPAkFHdFcizdOOBYwk_pL6pUZiy9ld24o-xBHIJZWcLmz_tMSFav77_9fbc2pPbwgnObR)\n\n![](https://lh5.googleusercontent.com/cbuvmjdXzUEgiSf6BL6jC3-TKAbwf1WASiNfypDXQcmeXLrr6OPhITsEjeEUo1dXtg3OW3YTPNez3XxjaRxeAV_Ox6dTq9XpbwNDqh2-iWiXqWhrgn2o1VIe45xKPGYsxxPluLkw)\n\n![](https://lh5.googleusercontent.com/wskI0Jy2g9icXoitcNDIp9VChhWlxu9fIDwepnqd0ds_Oq1m8Yt7ZHY8GZ_DvfS8IQcNfsJyBanCPIpp_GUznnzK3b3YBP7F2oV9gsi9k9WyKfwmWGCI4um2SKeNveXy2uaeD-sd)\n\n![](https://lh5.googleusercontent.com/zeDLw8eZvkjgSfWnUwOkIG_Ze3GmKAwrTH59o37K8XEMXDMcAOtUsOSzfEwP8qF7qgVHEbgCs6YbXyE9loDTZOX4q5JeRe2J4JMMufrD0H6wr-ADk9sMBzRRwgi3iRVjhTCLh0Uy)\n\n### **Check-out the popular hashtags :**\n\n[#next18](https://twitter.com/search?q=next18) [#googlenext18](https://twitter.com/search?q=googlenext18) [#knative](https://twitter.com/search?q=knative) [#kubernetes](https://twitter.com/search?q=kubernetes) [#istiomesh](https://twitter.com/search?q=istiomesh)\n\nAlso check out the keynotes:Keynotes from last week’s[ #GoogleNext18](https://twitter.com/hashtag/GoogleNext18?src=hash) here →[ http://g.co/nextonair](https://t.co/mVuwk0hw4i)\n","notHasFeatureImage":false,"slug":"recap-of-google-next’18"},{"id":86,"title":"Designating OpenEBS pods to run on specific K8S Nodes","author":"Ajesh Baby","author_info":"Product Manager at MayaData","date":"30-07-2018","tags":["OpenEBS","Kubernetes","Solutions","Scheduler","Scheduling"],"excerpt":"OpenEBS does not have a separate scheduler used to manage scheduling pods. Instead, it uses the Kubernetes scheduler for managing the scheduling needs of an administrator.","content":"\nOpenEBS does not have a separate scheduler used to manage scheduling pods. Instead, it uses the Kubernetes scheduler for managing the scheduling needs of an administrator. Kubernetes provides the following methods for controlling the scheduling pods on cluster nodes:\n\n- [nodeSelector](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector)\n- [Taints and Tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)\n\nFor more details and information about these features, you can refer to the Kubernetes documentation.\n\nIn this post, I would like to cover the different aspects of how to restrict/control the OpenEBS pods scheduling to a set of specific nodes in the Kubernetes cluster.\n\nOpenEBS deals with many types of Kubernetes pods throughout its life cycle. These can be broadly categorized into two types, control plane pods and data plane pods. Control plane pods are installed as part of installation of the following OpenEBS components:\n\n- OpeEBS API Server\n- OpenEBS Provisioner\n- OpenEBS Snapshot Controller\n\nData plane pods are installed as part of volume provisioning:\n\n- Target Pod\n- Replica Pods\n\nFor details on the exact steps of scheduling, see the configuration section [here](https://docs.openebs.io/docs/next/scheduler.html?__hstc=216392137.e7b2938c542eaf0f98426e5d8be4aa84.1579859056424.1579859056424.1579859056424.1&__hssc=216392137.1.1579859056424&__hsfp=3765904294).\n\nUse case: Let’s consider a scenario in which you have 20 nodes named Node1, Node2 ... Node20. You may want to designate Node1, Node2, Node3 as storage nodes so that all storage pods are scheduled only on these nodes.\n\nSolution: You can use Kubernetes scheduling methods to achieve this. Below are some of the possible options and their effect on scheduling pods to respective nodes.\n\n![Storage Pods](/images/blog/storage-pods.png)\n\nYou may select and use any of the above options based on your unique requirements.\n\nOption 2 does not necessarily guarantee storage pod scheduling on Node1, Node2 and Node3.\n\nOption 1, Option 3, and Option 4 will limit the scheduled OpenEBS pods to Node1, Node2 and Node3. Option 3 is my preferred choice, for the following reasons:\n\n- Other application pods will not be scheduled on my storage nodes, whereas Option 1 does present the possibility of other application pods being scheduled on Node1, Node2, and Node3.\n- While scaling the cluster for application deployments, I do not have to worry about changing the policy for storage. If I use option4, I must taint the new nodes with respective applications.\n- In this scenario, I am worried only about storage nodes, as these have local disks attached to them. I am not restricted to schedule an application pod deployment on any nodes other than storage nodes.\n","notHasFeatureImage":false,"slug":"designating-openebs-pods-to-run-on-specific-k8s-nodes"},{"id":87,"title":"OpenEBS 0.6 serves IOs amidst Chaos and much more","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"20-07-2018","tags":["DevOps","Kubernetes","OpenEBS","Stateful Containers","Storage"],"excerpt":"We are very excited to announce the availability of OpenEBS version 0.6. I would like to take a few minutes to talk about what has been keeping us busy in making it GA.","content":"\nWe are very excited to announce the availability of OpenEBS version 0.6. I would like to take a few minutes to talk about what has been keeping us busy in making it GA. We have been making a number of additions to both the design and code in the last few months. In this blog I’ll talk about:\n\n- Quality leveraging Chaos Engineering\n- Framework (CAS Templates) for supporting multiple storage engines\n\nI won’t talk about our new Storage Engine (cStor) — written in C — which is _almost_ ready. I’ll save that for a later blog.\n\nBefore going into specifics, I would like to express my sincerest gratitude to the OpenEBS community users and developers who are helping to make OpenEBS the most simple and easy to setup containerized attached storage (CAS) solution available for Kubernetes today — and the most popular open source one as well.\n\nSince the OpenEBS 0.5 release we have seen so many ways users have deployed OpenEBS many of which we had not envisioned when we started OpenEBS back in 2016. We are working hard to listen to the growing user community. Along these lines, we have a survey that is running through the end of July that takes 2–3 minutes that has proven helpful; please do fill it out if you have not already and we will even send you some swag: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\nThe timing of OpenEBS 0.5 was perfect in that it coincided with a take-off in interest in stateful workloads on Kubernetes. Some deployment patterns I’ve encountered just in the last few weeks include:\n\n- GitLab for internal IT teams\n- Kube weekly just featured a step by step blog on this subject: [https://openebs.io/blog/git-freedom-on-kubernetes](/blog/git-freedom-on-kubernetes)\n- Data Science training sessions\n- Here we are seeing hundreds of pods with stateful workloads spun up and destroyed repeatedly — really a great use case for container attached storage\n- Running Minio on OpenEBS which some users have called a happy marriage between OpenEBS block and Minio S3\n- We remain huge fans of Minio and are looking forward to more community led collaboration with our almost neighbors\n- OpenEBS being deployed as a basic part of DBs on Kubernetes; in particular we are seeing a good amount of NuoDB on OpenShift for example\n- The elastic SQL technology of NuoDB seems to resonate with lots and lots of users; we’re pretty proud that using OpenEBS underneath is becoming a common pattern\n- And of course many Containers as a Service offerings now include OpenEBS as a default option with more to be announced shortly\n\nAnd all this adoption means heterogeneity and dynamism.\n\n## Challenge 1: Kubernetes is resilient amidst Chaos and so must be storage\n\nBecause OpenEBS is deployed on so many varieties of Kubernetes and our fundamental job is to keep the data safe no matter what — we have been investing heavily in our ability to create these varied environments and their behaviors and to then measure and validate the resilience of OpenEBS as these environments respond to outages and increased load and so forth. We are seeing OpenEBS deployed across lots of varieties including:\n\n- Native Kubernetes or using Rancher, OpenShift, IBM Cloud Private, GKE, Tectonic, StackPoint Cloud, and others\n- Operating Systems -Ubuntu, CentOS, CoreOS and others\n- Pod Networking of all types, with Flannel being a favorite\n- Various cloud services — AWS EC2 remains the preferred option, with GCE growing in adoption amongst OpenEBS users\n\nEach combination comes with nuances that are unique and sometimes annoying as well. For example, recently a user on a Cloud Provider saw their nodes shut down frequently and occasional high network latency or packet drops in inter pod networking. Anyone with experience working with Storage Systems knows how detrimental these situations can be for latency-sensitive Stateful Applications.\n\nWe consciously chose the well-understood and widely used iSCSI protocol as the underlying storage connectivity used by Applications to connect to OpenEBS Volumes. There are many benefits to this architecture, but I will not address those here.\n\nThere are some annoying pieces when running iSCSI at scale as well. For example, depending on the response from the iSCSI targets and your operating system, there are some quirky things that can happen with iSCSI. The most notorious of these happens when iSCSI backed volumes move into read-only if you are using ext4 under certain conditions. You must then go through the steps for manually recovering the volumes. To address this, we have put together a troubleshooting guide that you can access [here](/docs/troubleshooting/volume-provisioning?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294).\n\nHowever, we wanted to solve as many of these issues as possible with the right approach. We stepped up our use of chaos engineering in our OpenEBS development process. We also extended and open sourced our in-house tooling, and we are starting to see it used more and more by engineers deploying stateful workloads on Kubernetes — whether or not OpenEBS is the underlying storage.\n\n## Solution : Chaos Engineered OpenEBS, the birth of Litmus.\n\nIf you would like an introduction to the Litmus project, which we open sourced at KubeCon in Denmark, visit the following link: [https://github.com/openebs/litmus](https://github.com/openebs/litmus)\n\nWe are also working on operators to add additional autonomic function into OpenEBS, leveraging improved metrics and advancements in CSI around node daemonsets and the mount-propagation feature. In the meantime, we use Litmus to increase automated real-world scenario testing to ensure improvements in every release. In this regard, a lot of effort has gone towards beefing up the tests that can simulate Chaos at Node, Network, Storage, RAM, and CPU. These typically contribute to Volume Pods switching nodes and, if not careful, interrupted IOs.\n\nOf course, this Chaos for Storage Application is something we believe should be applied to stateful workloads and underlying storage both during testing and as a part of a healthy chaos engineering practice. This is what led us to Open Source Litmus.\n\nOne outcome of our improved chaos engineering and testing is improvements to the resiliency of intra OpenEBS deployment communication. Specifically, we added enhancements to the responses sent by the iSCSI Target to the initiator; overall, this makes OpenEBS more resilient even when Pods are rescheduled unexpectedly and when the environment otherwise changes. You can learn more about these issues in the [release notes](https://github.com/openebs/openebs/releases/tag/v0.6).\n\nWe expect that the incidence of read-only issues will decrease greatly for the tested scenarios. We are constantly adding more scenarios, workloads and other tooling into Litmus to bolster the Jiva storage engine and other engines to come. Contributions to Jiva are of course always welcome!\n\n## Challenge 2: The evolving state of the State in Kubernetes!\n\nIf you regularly monitor storage developments, you will notice that Kubernetes is moving towards CSI and Snapshots are beginning to become a standard. There are enhancements to support Block Volumes and Topology aware Scheduling for Stateful Applications powered by Local PVs, which also benefits other PV types like OpenEBS.\n\nTo give an example, OpenEBS strongly prefers the case when the OpenEBS Controller Pod (and the application Pod) schedule on nodes where the OpenEBS Replica Pod resides. Currently, we achieve this via Pod/Node Affinity parameters. However, with Topology Aware scheduling, the constructs of pinning are efficiently done via the PV topology parameters.\n\nThat is just one example of new capabilities that we must now embrace. Features in Kubernetes now transition “quickly” from alpha to beta and the new paradigms/patterns that enter into Kubernetes must be adopted, or you will soon become outdated like the Third Party Resources (TPRs). However, we are not complaining about the pace of progress and continually contribute upstream to Kubernetes itself. We always seek to lend a hand to make Kubernetes an even better platform for storage and stateful workloads.\n\nNonetheless, the challenge remains. After all, the core of the Orchestration layer in OpenEBS is to deploy and operate the Container Attached Storage solution using Kubernetes native constructs. And the constructs just keep changing!\n\nLooking at the situation, we decided to step back and think about an architecture that would allow us to minimize the need to make code changes every time Kubernetes changes. For example, some users want to deploy their OpenEBS by specifying Pod Disruption Budges (PDBs) or setting specific Resource Limits for PVs in certain namespaces/users etc. We wondered: how can we embrace these new knobs, settings, and advances without endless code churn? This type of work — effectively upgrading the transmission of the underlying orchestration of OpenEBS — is not easy to do unless you really understand the architecture of OpenEBS. That’s not good — what’s the point of being open source if the code itself is too hard to work with and adjust? Fortunately, Kubernetes has CRDs, which provides a way forward.\n\n## Solution : Provide templates to Cluster Owners to define and manage the storage infrastructure.\n\nIn the OpenEBS 0.6 release, we have utilized the power of Kubernetes CRDs to provide a workable solution to introduce pluggable storage engines. OpenEBS now provides a complete workflow for developers and cluster administrators to choose the right storage software and hardware for their unique requirements. The control of the storage infrastructure stays with the cluster owner, and the ability to address a given need in storage lies with the developer. OpenEBS 0.6 brings the initial version of CAS Templates, which are YAMLs that can be scripted by cluster owners, fit into your GitOps, and are associated with Storage Classes.\n\nWe like the way OpenEBS CAS Templates are shaping up, and we can see many of the cluster owners’ needs being met over time, including enforcing of policies using tools like OPA. I will share more on this in upcoming blogs, but you can glance at them by reading this introductory documentation [here](/docs/concepts/cstor?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294). We intend to build upon this improved architectural pattern to do even more than the pluggability of storage engines. As always, we would especially welcome your feedback and use cases.\n\n### And there is more…\n\nYou will notice when you look at the release notes or try OpenEBS 0.6 that there are many other enhancements, including:\n\n- Configuring of OpenEBS for running stateful workloads that span across Availability Zones\n- Enabling the management of Snapshots and Clone from kubectl\n- Enhancement to mayactl to display volume status\n- Improved Integration and Unit Testing coverage\n- Enhanced Contributor Guides\n\nAnd, most importantly, product documentation has been overhauled to provide accessible insights about OpenEBS as well as a process to provide feedback.\n\nAs mentioned above, our next release also enables users to try out _cStor — a storage engine_ that is more efficient in terms of performance and capacity management. It also reduces the number of containers required to run OpenEBS. If you are interested in taking a look, please get in touch as we have some alpha users of cStor now.\n\nWith its strong community of users, developers, and partners building us into their solutions, it feels like OpenEBS is nearly unstoppable. As always, we look forward to your feedback and suggestions on this release and the direction that you want to see OpenEBS move going foward. Please reach out to us on Slack or add comments below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294)/\n\n[Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":false,"slug":"openebs-0.6-serves-ios-amidst-chaos-and-much-more"},{"id":88,"title":"How do I run a litmus test to compare storage performance on Kubernetes","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"16-07-2018","tags":["Benchmarking","Kubernetes","Litmus","OpenEBS","Solutions","Chaos Engineering"],"excerpt":"Ever so often, developers and devops engineers building or managing stateful applications on Kubernetes are on the lookout for for suitable storage options which serves their application’s specific needs.","content":"\nThis article belongs to a #HowDoI series on Kubernetes and Litmus\n\nEver so often, developers and devops engineers building or managing stateful applications on Kubernetes are on the lookout for for suitable storage options which serves their application’s specific needs. The emphasis could be on high-availability, provisioning ease, performance etc.., **Litmus** (as detailed in this [article](/blog/litmus-release-a-chaos-monkey-on-your-kubernetes-stateful-workloads)), is an attempt to arm them with the necessary info to make the right choice. One of the important storage tests is to simulate application workloads or multiply its effect using synthetic workload generators like fio. In this article, we list the steps to run a fio-based benchmark test using litmus\n\n![Evaluating storage performance w/ Litmus](https://cdn-images-1.medium.com/max/800/1*zRIZ9WjL7S0wq6Sp_IbzCw.png)\n\n## PRE-REQUISITES\n\n- At least a single-node Kubernetes cluster with the necessary disk resources, mounted on the node. (**_Note_**: _Certain storage solutions need minimum Kubernetes versions from which they are supported. For ex: Local PVs are beta from 1.10, OpenEBS needs 1.7.5+_)\n- Storage operator installed (typically, this includes control-plane elements like the static/dynamic provisioners, storage classes and other elements) with appropriate references to the node & disk resources (**_For example_**: _This may involve storage pool creation OR updating disk and node details in the static provisioners etc.,_)\n\n## STEP-1: Setup Litmus essentials on the Kubernetes cluster\n\n- Obtain the Litmus Git repository via a Git Clone operation on the Kubernetes master/Control machine used to manage cluster & set up the Litmus namespace, service account & clusterrolebinding by applying _rbac.yaml_\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ git clone https://github.com/openebs/litmus.git\n\nCloning into 'litmus'...\n\nremote: Counting objects: 2627, done.\n\nremote: Compressing objects: 100% (16/16), done.\n\nremote: Total 2627 (delta 2), reused 9 (delta 2), pack-reused 2609\n\nReceiving objects: 100% (2627/2627), 10.50 MiB | 4.23 MiB/s, done.\n\nResolving deltas: 100% (740/740), done.\n\nkarthik_s@cloudshell:~ (strong-eon-153112)$ cd litmus/\n\nkarthik_s@cloudshell:~/litmus (strong-eon-153112)$ kubectl apply -f hack/rbac.yaml\n\nnamespace \"litmus\" created\n\nserviceaccount \"litmus\" created\n\nclusterrole \"litmus\" created\n\nclusterrolebinding \"litmus\" created\n```\n\n- Create a configmap resource out of the cluster’s config file, typically at _~/.kube/config_, _/etc/kubernetes/admin.conf_ or elsewhere depending on the type of cluster or setup method\n\n(**Note**: _Copy the config file to admin.conf before creating the configmap out of it, as the litmus job expects this path_)\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ kubectl create configmap kubeconfig --from-file=admin.conf -n litmus\n\nconfigmap \"kubeconfig\" created\n```\n\n## STEP-2: Update the Litmus test job as per need\n\nThe litmus fio test job allows the developer to specify certain test parameters via ENV variables, such as the following:\n\n- The litmus fio test job allows the developer to specify the storage provider (PROVIDER_STORAGE_CLASS) and the node on which to schedule the application. (APP_NODE_SELECTOR)\n- The desired fio profile can also be specified. Currently, litmus supports simple [test-templates](https://github.com/ksatchit/litmus/tree/fio_test/tools/fio/templates), and is expected to grow to include multiple standard profiles. (FIO_TEST_PROFILE)\n- Certain simple test parameters such as the size of the test file (FIO_SAMPLE_SIZE) and duration of I/O (FIO_TESTRUN_PERIOD) can be specified as well, while the core I/O params continue to be housed in the templates.\n- The developer can choose to specify a comma-separated list of pods whose logs need to be collected for analysis of results, as well as the logs’ location on the host in the spec for the logger.\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ cd litmus/tests/fio/\n\nkarthik_s@cloudshell:~/litmus/tests/fio (strong-eon-153112)$ cat run_litmus_test.yaml\n\n***\n\napiVersion: batch/v1\n\nkind: Job\n\nmetadata:\n\nname: litmus\n\nnamespace: litmus\n\nspec:\n\ntemplate:\n\n metadata:\n\n name: litmus\n\n spec:\n\n serviceAccountName: litmus\n\n restartPolicy: Never\n\n containers:\n\n - name: ansibletest\n\n image: openebs/ansible-runner\n\n env:\n\n - name: ANSIBLE_STDOUT_CALLBACK\n\n value: log_plays\n\n\n - name: PROVIDER_STORAGE_CLASS\n\n value: openebs-standard\n\n\n\n - name: APP_NODE_SELECTOR\n\n value: kubeminion-01\n\n\n - name: FIO_TEST_PROFILE\n\n value: standard-ssd\n\n\n - name: FIO_SAMPLE_SIZE\n\n value: \"128m\"\n\n\n - name: FIO_TESTRUN_PERIOD\n\n value: \"60\"\n\n\n command: [\"/bin/bash\"]\n\n args: [\"-c\", \"ansible-playbook ./fio/test.yaml -i /etc/ansible/hosts -v; exit 0\"]\n\n volumeMounts:\n\n - name: logs\n\n mountPath: /var/log/ansible\n\n tty: true\n\n - name: logger\n\n image: openebs/logger\n\n command: [\"/bin/bash\"]\n\n args: [\"-c\", \"./logger.sh -d 10 -r fio,openebs; exit 0\"]\n\n volumeMounts:\n\n - name: kubeconfig\n\n mountPath: /root/admin.conf\n\n subPath: admin.conf\n\n - name: logs\n\n mountPath: /mnt\n\n tty: true\n\n volumes:\n\n - name: kubeconfig\n\n configMap:\n\n name: kubeconfig\n\n - name: logs\n\n hostPath:\n\n path: /mnt\n\n type: Directory\n```\n\n## STEP 3: Run the Litmus fio test job.\n\nThe job creates the Litmus test pod, which contains both the test runner as well as the (stern-based) logger sidecar. The test runner then launches an fio test job that uses a persistent volume (PV) based on the specified storage class.\n\n```\nkarthik_s@cloudshell:~/litmus/tests/fio (strong-eon-153112)$ kubectl apply -f run_litmus_test.yaml\n\njob \"litmus\" created\n```\n\n## STEP 4: View the fio run results.\n\nThe results can be obtained from the log directory on the node in which the litmus pod is executed (By default, it is stored in _/mnt_). The fio & other specified pod logs are available in a tarfile (\\_Logstash**.tar\\_\\_).\n\n```\nroot@gke-oebs-staging-default-pool-7cc7e313-bf16:/mnt# ls\n\nLogstash_07_07_2018_04_10_AM.tar hosts systemd_logs\n```\n\nThe fio results are captured in JSON format with job-specific result sections. Below is a truncated snippet reproduced from the log for a sample basic rw run:\n\n```\n{\n \"jobname\": \"basic-readwrite\",\n \"groupid\": 0,\n \"error\": 0,\n \"eta\": 0,\n \"elapsed\": 61,\n \"read\": {\n \"io_bytes\": 28399748,\n \"bw\": 473321,\n \"iops\": 118330.31,\n \"runtime\": 60001,\n \"total_ios\": 7099937,\n \"short_ios\": 0,\n \"drop_ios\": 0,\n \"slat\": {\n \"min\": 0,\n \"max\": 0,\n \"mean\": 0,\n \"stddev\": 0\n },\n \"write\": {\n \"io_bytes\": 28400004,\n \"bw\": 473325,\n \"iops\": 118331.38,\n \"runtime\": 60001,\n \"total_ios\": 7100001,\n \"short_ios\": 0,\n \"drop_ios\": 0,\n \"slat\": {\n \"min\": 0,\n \"max\": 0,\n \"mean\": 0,\n \"stddev\": 0\n }\n }\n }\n}\n```\n\n## CONCLUSION\n\nHow is this different from doing an fio package installation on the Kubernetes nodes and running tests?\n\n- Running an fio Kubernetes job will offer better control to simulating actual application loads when used with resource limits.\n- The litmus fio jobs with various profiles can be included as part of a larger suite using the executor framework, thereby obtaining results for different profiles.\n- Litmus (as it continues to mature) will provide jobs that perform Chaos tests against storage while running different types of workloads. Running a fio job lends itself to that model.\n- Finally, it is a more “Kubernetes” way of doing things!\n\nLet us know your experience with using fio-based performance tests with Litmus. Any feedback is greatly appreciated!\n","notHasFeatureImage":true,"slug":"how-do-i-run-a-litmus-test-to-compare-storage-performance-on-kubernetes"},{"id":89,"title":"Kubernetes storage extensions to Weave Scope","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-6-2018","tags":["Open Source","Weave Scope","Node Disk Manager","Persistent Volume","Kubernetes"],"excerpt":"It was in Austin KubeCon 2017 that I first got a deep look at Weave Scope, and could not stop falling in love with it. The visualisation Scope provides into Kubernetes resources is simply amazing.","content":"\nIt was in Austin KubeCon 2017 that I first got a deep look at Weave Scope, and could not stop falling in love with it. The visualisation Scope provides into Kubernetes resources is simply amazing. It greatly simplifies the tasks of an Administrator in dealing with the clutter of Kubernetes components and helps directly go to the component of interest and start observing and managing it.\n\nBeing tasked with the goal of simplifying storage management for Kubernetes, my immediate thought was, why can’t we use Scope for Kubernetes storage? Of course, storage in Kubernetes is a developing area and new features are always coming but the existing adoption of Kubernetes persistent storage volumes(PVs) concept was already pretty large and we thought it warranted extensions to Scope to include PVs.\n\nSo we got to and with the help of [Alexis](https://twitter.com/monadic) and the Weave team — we started coding!\n\nWe set out multiple milestones for this journey:\n\n- The first one — get the persistent volumes (PVs), persistent volume claims (PVCs) and Storage Classes (SCs) into Scope\n- The second one — add snapshot/clone support and start monitoring the volume metrics\n- The third one — bring in the disk or SSD or similar as a fundamental resource that is being managed by the Administrator just like they might want to sometimes take a look at CPU and Memory\n\n## Persistent Volumes (PVs)\n\nMost of the time, Persistent Volume Claims (PVCs) are the entry points to increasing the storage. The number of PVCs will be about the same as the number of pods, or slightly less in a reasonably-loaded Kubernetes cluster. The administrator will benefit from having visibility of which POD is using which PVCs and the associated storage classes and PVs. This is especially true if they are using the storage capacity of the Kubernetes clusters themselves. Adding this visibility is precisely is what we did to start.\n\n![PVC-PV-SC-POD Relationship on Scope](/images/blog/pvc-pv-sc-pod.png)\n\nYou can see this new visibility in Scope by using the newly-created filter “Show storage/Hide storage” under the PODs section. This filter puts the storage components in perspective with the remaining pods and associated networked-data connections. Users can **Hide storage** when not interested, or to reduce clutter.\n\nWe received an enthusiastic welcome to the Scope community from the Weaveworks team. We also found encouragement from [Alexis](https://twitter.com/monadic) and plenty of technical help from [Bryan](https://twitter.com/bboreham) at Weaveworks. The first pull request (PR) was really about adding PV, PVC and Storage Class support, and was merged into the Weave Scope master recently ([https://github.com/weaveworks/scope/pull/3132](https://github.com/weaveworks/scope/pull/3132) ).\n\n![PV-PVC-SC Integration into Scope](https://blog.mayadata.io/hubfs/0_iYXgl-m8oxyXVs1s.gif)\n\n## Future work:\n\n### Snapshots and Clones\n\nCI/CD pipelines are the most active areas in which DevOps are finding stateful applications on Kubernetes to be immediately applicable. Storing the state of a database at the end of each pipeline stage, and restoring them when required, is a commonly performed task. The state of the stateful application is stored by taking snapshots of its persistent volumes and is restored by creating clones of persistent volumes. We believe that offering visibility and administrative capabilities to manage snapshots and clones in Scope is a natural next step.\n\n### Disk Management and Monitoring\n\nHyper-converged Infrastructure (HCI) has yet to find its rhythm with Kubernetes, largely due to a lack of fully-developed tools for disk management and monitoring. Kubernetes now has a well-accepted method to provision and manage volumes and attach them to disk management. Therefore, the enabling of HCI for Kubernetes will be improved by new tools such as [Node Disk Manager (NDM)](https://github.com/openebs/node-disk-manager), to which, incidentally (humble brag), we are also contributing. With Disk being the fundamental component for storage and the main participant in the chaos engineering of storage infrastructure, it helps to have it visualised and monitored in a proper way. In large Kubernetes clusters (100+) nodes, the disks will be in the thousands. Scope’s resource utilisation panel is a powerful tool that brings in the visibility of CPU and Memory utilisation at the Host, Container and Process level. This is a natural extension to add Disk Capacity, Disk performance (IOPS and throughput) to this resource utilisation tool. Our view is shown in the figure below, that Disk performance can be added.\n\n![Current View of the Resource Utilisation Tool on Scope](https://blog.mayadata.io/hubfs/0_9SozVWeQ2F69fDQO.gif)\n\nAnother important aspect of disk management is simply browsing from the application volume all the way to the disk where the data is stored. It is not possible to locate the actual disk of a persistent volume if the underlying storage is a cloud-disk such as EBS or GPD, but if it is a Kubernetes local PV or OpenEBS volume, the volume data vs. physical disks relationship can be identified. This will be useful while managing the hyper-converged infrastructure on Kubernetes.\n\n![(Future work) PODs/Disks and Nodes Relationship at Scope](https://blog.mayadata.io/hubfs/0_WJA8ii6NlaBoS94H.gif)\n\nThe above screens are a dirty implementation on a dev branch that is still in process. However, it provides a good, quick glimpse of how a POD’s volume is linked to the associated disks.\n\n_Weaveworks team recently started community meetings led by [Fons](https://twitter.com/2opremio), and it appears to be a great beginning of broader community involvement into the development of Scope. You can access the public meeting notes at_\n\n_[https://docs.google.com/document/d/103_60TuEkfkhz_h2krrPJH8QOx-vRnPpbcCZqrddE1s/edit?usp=sharing](https://docs.google.com/document/d/103_60TuEkfkhz_h2krrPJH8QOx-vRnPpbcCZqrddE1s/edit?usp=sharing)_\n\n## Summary:\n\nWeave Scope is a very useful tool for Kubernetes administrators for visualising and basic administration. With the addition of extensions being added, and a wider community being formed, Scope’s adoption will certainly increase and benefit the Kubernetes eco-system. We are looking forward to being an active contributor to this excellent visualisation tool.\n\nPlease provide any feedback here or in the next Scope community meeting. We will be there!\n\nThanks to [Akash Srivastava](https://medium.com/@srivastavaakash?source=post_page) and [Satyam Zode](https://medium.com/@satyamz?source=post_page).\n","notHasFeatureImage":false,"slug":"kubernetes-storage-extensions-to-weave-scope"},{"id":90,"title":"OpenEBS announces the availability of version 1.0","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-06-2018","tags":["CNCF","Kubernetes","OpenEBS","Storage"],"excerpt":"In this blog, I will touch upon what I have learned from users about Open Source CAS solutions like OpenEBS at KubeCon EU 19, followed by a quick note on the major changes in 1.0 and what is in the immediate roadmap.","content":"\nCongratulations and thanks to each one of you from the OpenEBS community for reaching this significant milestone!\n\nIn this blog, I will touch upon what I have learned from users about Open Source CAS solutions like OpenEBS at KubeCon EU 19, followed by a quick note on the major changes in 1.0 and what is in the immediate roadmap.\n\nBeing an active member of the OpenEBS Slack channel and from the usage statistics, I was cognizant of the vibrant user community. For instance, a recent usage report generated by MayaData since 0.8 was released, OpenEBS is running on — 340 different flavors of Operating Systems, 90 different flavors of Kubernetes and across 75 different countries.\n\n![OpenEBS Usage Report](https://cdn-images-1.medium.com/max/800/0*QKRCQN6eguXuHH6u)\n\nKubeCon EU 19, was by far the most interesting for me — to interact with some of the above users who are already running OpenEBS in production or in the process of evaluation. For those of you, curious about the kind of views expressed from users and the partner community about OpenEBS and Kubernetes Storage at large, here are some interesting ones:\n\n**Feedback from Users running OpenEBS in production.** The choice to adopt OpenEBS was made after evaluating and comparing other available solutions as can be seen from several stories shared by the users:\n\n![User feedback about adoption of OpenEBS](/images/blog/slack-snnipets.png)\n\nThis growing adoption is attributed to the following:\n\n- The major one being the architectural superiority of using Kubernetes to build highly portable distributed systems — and **not locking** all the applications into a single point of failure — by making them write into storage system run by some other group or provider. OpenEBS helps to build a storage solution where each application gets its own storage controllers. This is a perfect validation of the Container Attached Storage Category, which is explained in more detail at a [CNCF Blog](https://www.cncf.io/blog/2019/05/16/a-year-later-updating-container-attached-storage/).\n- Containerized distributed applications are much smarter than their legacy server-based counterparts and these new containerized applications don’t have the same dependency on the storage system as before. What applications need are **ala-carte** of Storage Solutions that OpenEBS offers as Storage Engines. Depending on the application needs, cluster administrators can decide what components of OpenEBS are really deployed. For example — does the cluster need to only have Local PVs or a full-fledged storage solution like cStor.\n- Of course the fact the OpenEBS itself is completely developed in user space and just runs on any platform easily. **No kernel taints required.** Yes, currently with 1.0 — there is a performance hit, but there are enough use cases where the current performance is good enough.\n\nWords just can’t express the joy of hearing directly from users on how OpenEBS has helped them. It is just amazing to see the diversity of users as well; from universities to financial corporations, startups to enterprises, fresher to seasoned SREs.\n\nPersonally, the most humbling moment for me at the KubeCon EU 2019, was when an end-user approached me just to thank for the efforts we have put into OpenEBS. He works at a University as a SysAdmin and he mentioned that using OpenEBS — he is able to spin up a self-contained Kubernetes Cluster with host storage that can be used by the stateful applications. This has helped in the number of support calls he receives from the University IT department when rolling out Stateful Applications that needed some hand-holding to provide the PVs.\n\nA similar sentiment is being expressed by large enterprises where the Infra team is supposed to roll out services to 100s of there application teams, in a more agile and uniform way than before. Some of the enterprises run Big Data pipelines at mammoth scale and have found OpenEBS to contains the right set of abstractions that can be extended and used.\n\n**Feedback from the growing community and partner ecosystem** that comprises of individual contributors, home users, enterprise solution architects, technology enthusiasts to hardware vendors included:\n\n- Solution Architects, DevOps, SREs >> I have put together a few scripts to make backups and restores with the Velero plugin for cStor volumes a little easier, since there are some parameters to remember and some additional steps required for both backups and restores other than using the basic Velero commands. I created a tool for [velero openebs](https://github.com/vitobotta/velero-openebs-backup) — with a few notes in a comment, in case anyone is interested\n- Storage Solution Engineers >> Hey, I know OpenEBS was not meant for this — but I think I made some changes can now serve Volumes from Kubernetes Cluster to Virtual Machines. A custom SDS, if you will, that is developed using OpenEBS.\n- Storage Vendors >> We are a storage devices company and OpenEBS can be used to run in an optimized way and help us shift storage boxes.\n- Kubernetes Managed Service Providers >> It will be cool to have OpenEBS available via the Operator Hub/Marketplace.\n\nBy the way, it is hard to gauge how much of the above discussions were really made possible by OpenEBS becoming part of the CNCF family. However, a major shift that I have observed since becoming a CNCF project, is that both end users and partners are more forthcoming to share their stories and integrate OpenEBS into their portfolio. _For anyone considering donating their projects to CNCF, I highly recommend it **—** there is a steep learning curve that helps with Governance of the project, by just being part of the efforts to propose the project into CNCF._\n\nClearly, these days the discussions at KubeCon EU are not around What or Why OpenEBS? but around When a specific feature is going to be made available. While the top requests with regards to missing features were around CSI support, automation of the operations like — disk provisioning and capacity expansion and performance improvements — they were really not blocking users from running OpenEBS in production. There are lots of users out there, that have found the current performance offered by the OpenEBS Data Engines was good enough for their use cases, and they have gone ahead and implemented playbooks and tools for performing maintenance and monitoring operations.\n\nPost KubeCon, having known and also met in person some of these early adopters that are running OpenEBS in production with a set of home-grown solutions, we felt a sense of responsibility to release 1.0 at the earliest and promise to provide long term support on the current feature set. As we progress with implementing additional functionality around operators, CSI, supporting the latest from Kubernetes Storage, and so forth we are also making a commitment that the current version will be supported.\n\nHence, for 1.0 — we changed our gears a bit to focus on helping address the feedback received on the existing feature sets for all the three data engines now supported by OpenEBS and deployed in production- Jiva, cStor Data Engines and Local PV.\n\nSome significant changes in 1.0 are as follows:\n\n**Enhanced Lifecycle Management of Block Devices**. The component of OpenEBS — NDM — Node Disk Manager or Node (Storage) Device Manager as we call it now, has been enhanced to support Block Device Claim (BDC). A new NDM Operator has been introduced in 1.0, that helps to request and reserve a Block Device before using that for either creating Local PV or cStor Pool. NDM is also being used independently of OpenEBS Storage Engines, and the ability to claim a Block Device — similar to PVC — enables the sharing of the block devices without stepping on each other.\n\nBlock Device(BD) is a new CR — which is a preferred way to represent a storage device than a Disk CR ( which was used in earlier releases.) Disk CR are also present — to allow for backward compatibility. The cStor Data engines and Local PVs that typically use block devices have been enhanced to use BDC/BD in place of Disk.\n\nWhen a BDC is deleted, the BD is released. And before it can be claimed by another BDC, the NDM takes care of deleting the data that has been written from the previous consumer. The cleaner utility used is derived from the Kubernetes Local PV cleaner jobs.\n\n**Enhance the OpenEBS Local PV to be used with Block Devices.** In the previous release, we introduced Local PVs that can be used with a hostpath, in this release, we have added support to create a Local PV directly on a Block Device (discovered by NDM). The Block Device can either be a raw block device (Example: GPD) — in which case it will be formatted (Example: Local SSDs on GKE) with the requested filesystem type or already mounted block device.\n\nAlso included in this release are:\n\n- Bug fixes reported around the cStor and Jiva data engines that surface after running OpenEBS on nodes that experience frequent reboots.\n- Additional debugging tools for cStor and Jiva engines, jivactl being the significant one that will help with clearing up of older snapshots. This tool has to be used to clear older snapshots to avoid hitting a cap limit of 255 snapshots in older releases.\n- Added few more Prometheus monitoring metrics\n- Added several e2e tests and included more applications into the OpenEBS CI pipeline maintained by [MayaData](https://mayadata.io/) at [openebs.ci](https://openebs.ci/)\n\nFor detailed changes summary, please check the [OpenEBS 1.0 Changelog](https://github.com/openebs/openebs/wiki/Change-Summary----v1.0).\n\nPlease refer to the [OpenEBS Documentation](https://openebs.io/docs?__hstc=216392137.ed92f0691adfb1cbf08ea329504224a3.1580116629364.1580116629364.1580116629364.1&__hssc=216392137.1.1580116629364&__hsfp=3765904294) to get started.\n\nOpenEBS 1.0 — ships with multiple Data Engines to choose from depending on your application needs:\n\n- OpenEBS Jiva PVs — can be used if your kubernetes nodes don’t have the capability to add raw block devices, but have extra capacity available on the host filesystem. This is the first and the longest running in production with lots of Kubernetes tunables available like — to customize the location where data is saved, the specific nodes on which data needs to be replicated — within or across availability zones, setup the volumes for thin provisioning, and so forth.\n- OpenEBS cStor PVs — is the preferred option when your nodes have raw block devices. cStor Data Engine continues to be a preferred solution for use cases that require instant snapshot and clone of volumes.\n- OpenEBS Local PVs — best suited for applications that can do their own replication and require high performance. OpenEBS Local PVs can work with either hostpath or with block devices that are already attached to the nodes.\n\nThere are a lot of options available to customize from the type of block devices that can be used for OpenEBS volumes to customizing the resources allocated to the Storage Pods.\n\nOpenEBS has turned a new leaf in its journey by releasing version 1.0. Thank you for all the support and love that you have shown along this journey. We are forever more committed to learn from you and help you realize the Data Agility that Cloud Native promises to offer.\n\nOur immediate focus is on getting OpenEBS ready for CSI and enhancing the OpenEBS Operators — for managing the Day 2 operations like dealing with Kubernetes upgrades, ASG, expansion of cStor Pool capacity or migrating them to new nodes and so forth. Development on these items is being tracked in the Kubernetes style at [Product Planning Sheet](https://docs.google.com/spreadsheets/d/1bbphUqbxShBhgr1VHaEQUzIGMaJJacPNKc1ckNXU1QE/edit).\n\nThanks to CNCF, we have the following mailing lists to connect apart from the [Slack](https://openebs.org/community).\n\n- For OpenEBS project updates — subscribe to [https://lists.cncf.io/g/cncf-openebs-announcements](https://lists.cncf.io/g/cncf-openebs-announcements)\n- For interacting with other OpenEBS users, you can either join the [Slack](https://openebs.org/community), raise issues on [Github](https://github.com/openebs/openebs/issues) or send an email to [https://lists.cncf.io/g/cncf-openebs-users](https://lists.cncf.io/g/cncf-openebs-users)\n\nAs always, I am eager to learn from you. Please hit me up on [twitter (@kiranmova)](https://twitter.com/kiranmova), [slack](https://openebs.org/community) or on here in the comments.\n\nThanks to [Murat Karslioglu](https://medium.com/@muratkarslioglu?source=post_page). [Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":true,"slug":"openebs-announces-the-availability-of-version-1.0"},{"id":91,"title":"Git freedom on Kubernetes","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"05-06-2018","tags":["DevOps","Git","Gitlab","Kubernetes","OpenEBS"],"excerpt":"Here is one of the fastest ways to get your private repository with Gitlab up and running on your Kubernetes environment — Let’s “Make DevOps lifecycle private again”","content":"\n#### Steps to easily run GitLab\n\nAfter Microsoft announced the acquisition of GitHub, many developers raised concerns on social media about Microsoft’s history of unsuccessfully running the acquired businesses like Skype, Nokia’s handset business, Navision and other 150 companies (you probably haven’t noticed) they have swallowed up over the years.\n\nOther than keeping the developer’s life-support plugged, one of the biggest concerns is that MS would use its power on GitHub repositories to analyze trends among software development in order to launch competing products. Fears that GitHub privacy may be in jeopardy have already led many developers to jump off the ship or consider alternatives. GitLab’s publicly available [status graphs](https://t.co/svpWpI0Rb2) show spikes of 70x increase in imported repositories (average 100 vs 7.5K), a validation of increased user apprehension.\n\nHere is one of the fastest way to get your private repository with Gitlab up and running on your Kubernetes environment — Let’s “**Make DevOps lifecycle private again**” ©\n\nCurrently, the easiest and recommended way to install GitLab on Kubernetes is using the [Gitlab-Omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.html) Helm charts.\n\nGitlab-Omnibus deploys every feature a small deployment would require including the [Container Registry](https://docs.gitlab.com/ee/user/project/coThis year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.ntainer_registry.html#gitlab-container-registry), [load balancer (NGINX)](https://github.com/kubernetes/ingress/tree/master/controllers/nginx), [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/), and [Runner](https://docs.gitlab.com/runner/).\n\n#### **Prerequisites**\n\nMinimum requirements for a multi-node cluster:\n\n**Hardware**\n\n- **Boot node:** 1x 1+ core(s) >= 2.4 GHz CPU, 4GB RAM, >=100 GB disk space\n- **Master node:** 1 or 3x 2+ cores >= 2.4 GHz CPU, 4+GB RAM, >=151 GB disk space\n- **Worker node:** 3x 2+ cores >= 2.4 GHz CPU, 4+GB RAM, >=100 GB disk space\n\nSince I’m not planning to run anything heavy, I’ll be using 3 nodes, and will install Master, Proxy, and Workers an all 3.\n\n**Software**\n\n- [Ubuntu 16.04 LTS](https://www.ubuntu.com/download/server) (RHEL 7.x is also supported)\n- Docker 1.12 to 17.03\n- Kubernetes 1.7+ Cluster (You can use [IBM Cloud Private 2.1.0.2](http://containerized.me/how-to-install-openebs-on-ibm-cloud-private/) or [Red Hat OpenShift Origin](http://containerized.me/how-to-install-openebs-on-openshift/))\n- [kubectl](https://github.com/kubernetes/kubectl)\n- Helm client\n- A [GitLab Omnibus](https://docs.gitlab.com/omnibus/) Pod, including Mattermost, Container Registry, and Prometheus\n- An auto-scaling [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor\n- [Redis](https://github.com/kubernetes/charts/tree/master/stable/redis)\n- [PostgreSQL](https://github.com/kubernetes/charts/tree/master/stable/postgresql)\n- [NGINX Ingress](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress)\n- [OpenEBS](https://github.com/openebs/openebs) persistent volumes for Data, Registry, Postgres, and Redis\n\nThe Kubernetes instructions described below using Helm are generic and should work on all other platforms.\n\n**Installing GitLab and OpenEBS using the Helm Chart**\n\nGitLab depends on stateful applications like Redis and PostgeSQL, and requires persistent volumes for its data and the registry. Here, I will simplify the storage provisioning using OpenEBS.\n\nFirst, install OpenEBS using the chart.\n\n```\nhelm install — name ‘openebs-gitlab-test’ stable/openebs\n```\n\nOptional: If you would like to customize your OpenEBS installation you can also use a copy of the [value.yaml](https://raw.githubusercontent.com/kubernetes/charts/master/stable/openebs/values.yaml) file from the OpenEBS chart and modify parameters listed [here](https://github.com/kubernetes/charts/tree/master/stable/openebs).\n\n```\nhelm install — name ‘openebs-gitlab-test’ -f values.yaml stable/openebs\n```\n\nNext, add the predefined storage classes.\n\n```\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml\n```\n\nThere are many ways to enable OpenEBS for use by GitLab. The fastest is by making one of the OpenEBS storage classes a default StorageClass:\n\nList available OpenEBS storage classes in your cluster.\n\n```\nmurat@icpnode1:~$ kubectl get sc\nNAME PROVISIONER AGE\nopenebs-cassandra openebs.io/provisioner-iscsi 18d\nopenebs-es-data-sc openebs.io/provisioner-iscsi 18d\nopenebs-jupyter openebs.io/provisioner-iscsi 18d\nopenebs-kafka openebs.io/provisioner-iscsi 18d\nopenebs-mongodb openebs.io/provisioner-iscsi 18d\nopenebs-percona openebs.io/provisioner-iscsi 18d\nopenebs-redis openebs.io/provisioner-iscsi 18d\nopenebs-standalone openebs.io/provisioner-iscsi 18d\nopenebs-standard openebs.io/provisioner-iscsi 18d\nopenebs-zk openebs.io/provisioner-iscsi 18d\n```\n\nEither create your StorageClass or pick one of the predefined classes. _openebs-standard_ creates 3 replicas and is an ideal candidate here to be used for most of the stateful workloads. Let’s mark this StorageClass as default.\n\n```\nkubectl patch storageclass openebs-standard -p ‘{“metadata”: {“annotations”:{“storageclass.kubernetes.io/is-default-class”:”true”}}}’\n```\n\nNo verify that your chosen StorageClass is indeed the **default**.\n\n```\nmurat@icpnode1:~$ kubectl get sc\nNAME PROVISIONER AGE\nopenebs-cassandra openebs.io/provisioner-iscsi 18d\nopenebs-es-data-sc openebs.io/provisioner-iscsi 18d\nopenebs-jupyter openebs.io/provisioner-iscsi 18d\nopenebs-kafka openebs.io/provisioner-iscsi 18d\nopenebs-mongodb openebs.io/provisioner-iscsi 18d\nopenebs-percona openebs.io/provisioner-iscsi 18d\nopenebs-redis openebs.io/provisioner-iscsi 18d\nopenebs-standalone openebs.io/provisioner-iscsi 18d\nopenebs-standard (default) openebs.io/provisioner-iscsi 18d\nopenebs-zk openebs.io/provisioner-iscsi 18d\n```\n\nNext, we can install the GitLab-ce chart. It is recommended to save your configuration options in a values.yaml file for future use.\n\n```\nwget https://raw.githubusercontent.com/kubernetes/charts/master/stable/gitlab-ce/values.yaml\n```\n\nEdit the _values.yaml_ file and at minimum, add the **externalUrl** field. Otherwise, you’ll end up with a non-functioning release.\n\nHere is how my _values.yaml_ file looks like after these changes:\n\n```\nimage: gitlab/gitlab-ce:9.4.1-ce.0\nexternalUrl: http://containerized.me/\nserviceType: LoadBalancer\ningress:\nannotations:\nenabled: false\ntls:\nurl: gitlab.cluster.local\nsshPort: 22\nhttpPort: 80\nhttpsPort: 443\nlivenessPort: http\nreadinessPort: http\nresources:\nrequests:\nmemory: 1Gi\ncpu: 500m\nlimits:\nmemory: 2Gi\ncpu: 1\npersistence:\ngitlabEtc:\nenabled: true\nsize: 1Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\ngitlabData:\nenabled: true\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\npostgresql:\nimageTag: “9.6”\ncpu: 1000m\nmemory: 1Gi\npostgresUser: gitlab\npostgresPassword: gitlab\npostgresDatabase: gitlab\npersistence:\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\nredis:\nredisPassword: “gitlab”\nresources:\nrequests:\nmemory: 1Gi\npersistence:\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\n```\n\nNow, install the chart.\n\n```\nhelm install — name gitlab-test -f values.yaml stable/gitlab-ce\n```\n\nList the pods and confirm that all pods are ready and running.\n\n```\n$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\ngitlab-test-gitlab-ce-dd69cdf4b-69vmb 1/1 Running 0 11m\ngitlab-test-postgresql-75bf9b667d-lwj2b 1/1 Running 0 11m\ngitlab-test-redis-998998b59-hzztj 1/1 Running 0 11m\nopenebs-gitlab-test-apiserver-68fc4488fd-jf8gz 1/1 Running 0 1h\nopenebs-gitlab-test-provisioner-7dfdf646d8–9wpmg 1/1 Running 0 1h\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-ctrl-74d4b59c9f-bjtg2 2/2 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-6ds26 1/1 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-99mbh 1/1 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-d8d4z 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-ctrl-bd7cff65f-ph8dr 2/2 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997–2lm4x 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997-jldjs 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997-kzlrc 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-ctrl-668f5988c5-hv8vb 2/2 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-hsn49 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-lj64g 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-z6kfd 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-ctrl-585cf7c97d-58pnq 2/2 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-5bzn6 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-9dz5f 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-snkfb 1/1 Running 0 11m\n```\n\nGet the list of persistent volumes.\n\n```\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-postgresql openebs-standard 17m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-redis openebs-standard 17m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-gitlab-ce-data openebs-standard 17m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800 1Gi RWO Delete Bound default/gitlab-test-gitlab-ce-etc openebs-standard 17m\n```\n\nYou can see above that four persistent volumes were created (**postgresql, redis, gitlab-ce-etc, gitlab-ce-data**), and each volume is protected by 3 replicas.\n\nNow go to the external endpoint address you have defined and start using GitLab after you set your new password.\n\n![](https://lh4.googleusercontent.com/9UnAe3ZuKt_weq1IbxOrgA_JQMpXE2ZCd80PgDxIodeUdFslr-wCt2DUjbWyoERYWa6RKht8JYvihV-_dQS0EYArc4dJhkPPtN0cGPNfYcDsiHgtjS7unCLOW9MTDre79AjZ660xm6IN94OPew)\n\nNow click on **Create a project**, then import your existing project from GitHub and start using GitLab.\n\n![](https://lh4.googleusercontent.com/CDe7SDXnBmCL5IGVTIOYATjzjZN2iMPsZaVBmuY3-l6qXFX8xReeU6M234eX0ELY1Pips6JfR1FJb4rzfL_d53KLDon0MrzBKqQvuBslQDboCw1yPehiKrSf771PMy79ckmPdLGWnhmijDFkVg)\n\n![](https://lh6.googleusercontent.com/AFqy2l5MohpC1kCk5k2yFoZA90qJGabfUymqmMmI0kFqcpgqXnrapoYCs1dMfrFDqKj-37ncNvoCe7Kf8UfCQq6VRvmFMK742abC58ju6TiRSUk2yeq1OtMBZWPMd3pqyQWawBDgUcSpSZ8Djg)\n\n---\n\nHopefully, this helps anyone who is motivated to reexamine their approach to Git to quickly and easily start running GitLab on Kubernetes. Thank you for reading, and please provide any feedback below or via Twitter — [**@**muratkarslioglu](https://twitter.com/muratkarslioglu)_Originally published at _[_Containerized Me_](http://containerized.me/git-freedom-on-kubernetes/)_._\n","notHasFeatureImage":false,"slug":"git-freedom-on-kubernetes"},{"id":92,"title":"Berlin K8s meetup retrospect","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"31-05-2018","tags":["OpenEBS","Kubernetes","DevOps"],"excerpt":"Last week, I was invited to give a talk about OpenEBS during the Kubernetes meetup in Berlin. The event was hosted by the friendly folks at Profitbricks, who once again I want to thank for the lovely venue and experience.","content":"\nLast week, I was invited to give a talk about OpenEBS during the Kubernetes meetup in Berlin. The event was hosted by the friendly folks at Profitbricks, who once again I want to thank for the lovely venue and experience. Matt Baldwin, from our friends at StackPoint Cloud, was once again an organizer — thank you Matt for your ongoing support of the community (and of OpenEBS as well).\n\nAs we went over some of the nitty-gritty details on how we are building OpenEBS, I received a few questions that I thought deserved an extended response. I will address one of those here.\n\nA common question was: “If I’m running on an __ system, what is the benefit of OpenEBS?”\n\n**_So, let’s dive into one of the biggest questions— what if you are running on AWS and you are using EBS volumes, why use OpenEBS on top?_**\n\nFirst of all, we already made the implicit assumption that if you were to run an application managed by Kubernetes, you are using EBS volumes. But, this assumption does not have to be true. In fact, if you want fast and performant storage, AWS suggests that you use optimized storage instances, meaning _not using EBS_ volumes at all.\n\nThe EBS volumes are not the fastest thing you can get from AWS, but they do provide replication across zones and snapshots. This brings me to the second part, a snapshot, in AWS, is stored in S3. That bill (depending on dataset size) can quickly and easily pile up with the number of snaps. With OpenEBS, we store the snapshot locally on the volume itself, and we don’t require you to “copy” this into S3 or a similar location. Also, consider the amount of time it takes for a snapshot to be created in AWS; it can take hours and the snapshot restores require you to attach a new EBS volume. Moreover, if you do these types of things, you have to “fiddle” with the OS to get it attached, something that OpenEBS handles for you, making snapshots instantly and allowing you to rollback immediately.\n\nThere is also the matter of granularity; with OpenEBS, you can control everything per application or even per workload. You can fine-tune the replication factors (or not have it all) and you can use just one EBS volume for all your apps since OpenEBS handles the slicing and dicing for you.\n\nSo, this was about using EBS volumes, I hope the advantages are apparent and easy to understand. This brings data agility to your applications, not just in AWS, but also across clouds: public, private or hybrid. The developer does not have to make any changes to his or her YAML, and can write once and run everywhere, precisely what containers are supposed to do. Do not let others take that freedom away from you =)\n\nBack to the non-EBS volumes, what AWS refers to as “instance store.” More information from Amazon:\n\n![Instance stores](/images/blog/instance-stores.png)\n\nAs the docs point out, when you use this, you get high speed with zero features. With OpenEBS, you get high speed and all of the features if you delegate control of the “instance store” to OpenEBS. An interesting note, the instance store comes with the price of the machine instance itself, so there are no extra charges on your bill.\n\nIn a future blog I’ll cover some other common questions that arose — and I’m also going to continue talking about the shift of IO and so forth to the user space, and what it may mean for Container Attached Storage and running IO intensive workloads in general.\n\nFeel free to ask any questions below, or maybe I’ll see you on the [OpenEBS slack community](https://slack.openebs.io/?__hstc=216392137.c2ca19ae0ec72f00a9ae3bf6f8a512a3.1579861713935.1579861713935.1579861713935.1&__hssc=216392137.1.1579861713935&__hsfp=3765904294) where we are discussing many related subjects.\n","notHasFeatureImage":false,"slug":"berlin-k8s-meetup-retrospect"},{"id":93,"title":"Container Attached Storage (CAS) — Taking off nicely","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-05-2018","tags":["CAS","Kubernetes","Stateful Applications","OpenEBS","MayaOnline"],"excerpt":"I had the fortune of presenting to a group of brilliant folks at SNIA India SDC event last week. This event being in Bangalore, I could sense the heat emanating from technology savvy brains mostly from the storage companies like DELL-EMC, NetApp, RedHat storage, HP storage etc","content":"\nI had the fortune of [presenting](https://www.slideshare.net/OpenEBS/openebs-cas-sdc-india-2018) to a group of brilliant folks at SNIA India SDC event last week. This event being in Bangalore, I could sense the heat emanating from technology savvy brains mostly from the storage companies like DELL-EMC, NetApp, RedHat storage, HP storage etc and the questions at the end of the presentation were a proof of it. Lots of great questions on the lines of “Nice architecture, I never knew storage can be run completely in user space”, “CAS — great topic, I understand the value and benefits, but are not you adding too many containers to the equation?”. In the interest of answering those questions in detail, I thought let me take the most commonly asked ones and answer them in this short article.\n\n## Just a quick recap on CAS:\n\n[Container Attached Storage (CAS)](/docs/concepts/cas) is a new storage architecture to run the entire storage software in containers and hence in user space. This architecture has many benefits, primary one being “a dedicated storage controller per application” and bring in the possibility of hardening the storage controller for a given application workload. Read more on the benefits at the [CNCF blog](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/). A typical CAS architecture example is shown below.\n\n![CAS architecture with controller and replica pods for each application](https://cdn-images-1.medium.com/max/800/1*4dJDmPbxxrP-fZK7NZZmYg.png)\n\nNow back to the CAS FAQ, here are a couple:\n\n### Q1: Well, how can you run entire storage in user space? I thought it can never be run in user level because of performance reasons.\n\nFor more technical answer, read our [CTO’s blog](/blog/the-mule-and-the-flash-going-for-a-run) on this precise topic 🙂\n\nShort answer is, we think storage should take advantage of container technology, so in CAS, we run storage as a microservice. The performance will be taken care by using technologies like SPDK, VPP and also taking advantage of the abundant availability of compute cores on the node.\n\nReality is that, linux kernel, as it is today, cannot deliver all the IOPS from the underlying NVMe flash disks. Do we think kernel can deliver 12 Million IOPS from the 24 NVMe disks ? Of course not, not today !!\n\nIn summary, higher performance can only be delivered using CAS architecture, SPDK, VPP and using more CPU cores.\n\n![CAS with SPDK leads to higher performance](https://cdn-images-1.medium.com/max/800/1*aKjepAaB5sIZF-hOq_dxIg.png)\n\n### Q2: In CAS, you are adding more containers to the cluster because of storage. Isn’t that increasing the compute needs of the Kubernetes cluster?\n\nCAS enables native hyper convergence capability on Kubernetes. You don’t need an external storage array to manage the storage/data needs of applications on K8S, thats a lot of saving of CPU and hardware. With CAS, the overall TCO reduces as the additional storage (local disks), CPU and software (CAS) are provisioned on the same K8S cluster nodes and avoids the need of expensive external storage arrays.\n\nAs for the the question of dealing with large number of containers is concerned, there are nice tools like Weave Scope which you can use on the local K8S.\n\n![Application relationship with storage (PVC, PV, SC, and CAS)](https://cdn-images-1.medium.com/max/800/1*RQYjI0MdsXf1kj8AGqLJZA.png)\n\n## Conclusion:\n\nWe continue to say — With CAS, storage fades away as a concern !! Join our [slack community](https://slack.openebs.io) to find more !!\n","notHasFeatureImage":false,"slug":"container-attached-storage-(cas)-—-taking-off-nicely"},{"id":94,"title":"My First Contribution to OpenEBS \\#OSS","author":"Vipul Gupta","author_info":"Enter a short bio? No, thank you. If you like to find me or my content, then I would be at http://mixstersite.wordpress.com/","date":"28-05-2018","tags":["OpenEBS","Go","Golang","Summer Hackfest","Open Source"],"excerpt":"Writing documentation for any project is tough, be it big or small, propriety or open-source. Rewriting and improving it is even tougher. Let no one tell you any different.","content":"\nWriting documentation for any project is tough, be it big or small, propriety or open-source. Rewriting and improving it is even tougher. Let no one tell you any different. Take it from someone who scoured through 6000+ lines of documentation written in Markdown spread across 20 or so files and been able to fix about 900 odd lines of them for a major open-source project. That’s how my story goes if you want to know about my first contribution to OpenEBS.\n\nHonestly, I have little experience in Go, I have never been inclined to use it much. As I am already comfortable, writing code in Python for about a year now. But that doesn’t stop me from learning new languages and technologies. Hence my utmost dedication to cure myself of the “Too Much too Learn” Syndrome. I went to the **Women Who Go meetup** in New Delhi, India for a [Go 101 workshop](https://twitter.com/vipulgupta2048/status/977893034808434689?s=09). And that’s how I got to know about the open-source project, OpenEBS, and their exciting Summer HackFest.\n\n[OpenEBS](https://openebs.io/) is an open source storage platform that provides persistent and containerized block storage for DevOps and container environments. OpenEBS is also tightly integrated with Kubernetes and is a part of Kubernetes-Incubator project. And works with all popular workloads such as PostgreSQL, Redis, Jenkins etc.\n\nNow, OpenEBS is still under active development from the team. Hence comes in, [Summer HackFest](https://openebs.io/hackfest). **Summer HackFest** is a unique opportunity for new contributors and people working with Go to develop and fix issues for OpenEBS. By contributing to the Hackfest, you get community access, useful knowledge on working with new technologies, learn new skills and along the way win cool goodies, swag. Also, the chance to take home the grand prize, a laptop. HackFest starts out in April 2018. Being open-source, all contribution was needed to be submitted to [GitHub](https://github.com/openebs/openebs). I did and oh man, I had a good time writing code for OpenEBS. Sounds good. Want to contribute?\n\n---\n\n## Let’s get you started\n\nRefer to this [quick guide](https://github.com/search?utf8=%E2%9C%93&q=org%3Aopenebs+is%3Aissue+label%3Asummerhack+label%3Akind%2Funit-test&type=) to get started. You would also need to be familiar with the basics of Go, hence you can refer to the widely recommended tutorial [https://tour.golang.org/](https://tour.golang.org/), for the same. After that, make sure to join the OpenEBS Slack channel to resolve all doubts and questions that you might have. The link can be found here in the [contributing document](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md) on their repository. Read the document (I re-wrote it) And you are committed.\n\nGo to their website that I linked above, and figure out which section of issues you would like to contribute. Once done. Head over there, and do your thing. Push the changes, make that PR and rest is history.\n\n## Well, sure thing but how was your experience?\n\n![Contributors to OpenEBS Hackfest](/images/blog/hackfest-contributors.png)\n\nMy experience has been great so far while contributing to OpenEBS (_I am still actively contributing to it_). Everything was a breeze. They have very flexible rules and easy guidelines, no hard checks with the CI. The reviewers are very helpful, and would assist you in every possible manner ( Shoutout to Kiran Mova !!) I have been working on solving issues in the documentation of OpenEBS and vagrant files that are being used in the project.\n\nEarlier, I also attended a webinar OpenEBS conducted on Introduction to contributing and solving issues of Maya and scope, which was very informative and helpful. Hence goes on to show their dedication to helping their contributors even further then most people would go. The codebase could be quite intimidating at first, but with time and the helpful answers provided by the very vibrant and active community will keep you going. I can tell you for a fact, that the journey is the reward here.\n\nI asked so many questions, and never got a blowback. I really appreciated all the support shown to me by the community. The environment is beginner-friendly and very welcoming, great for all new contributors. It couldn’t get any better with the incentives they add to it for contributing.\n\nI honestly had a lot of fun while contributing. One of the most notable [pull requests](https://github.com/openebs/openebs/pull/1511) of mine to Summer Hackfest was about rewriting and improving the documentation, which received a whopping 392 comments. They were sweet enough to put my name on the website with the fellow contributors and made me the star contributor of the week. Woohoo !! They also awarded me with a book as a token of their appreciation. Thanks, OpenEBS !!\n\nAnd that’s it for this time, I guess. For all my PR’s to Summer HackFest, [click here](https://github.com/search?utf8=%E2%9C%93&q=org%3Aopenebs+is%3Apr++label%3Asummerhack++author%3Avipulgupta2048+is%3Amerged&type=Issues). I will update the blog post with a picture of the goodies once I receive them (_So excited_). I highly recommend heading over to [https://openebs.io/hackfest](https://openebs.io/hackfest), to take part in this golden opportunity to contribute to such a brilliant project and being a part of their community.\n\n---\n\nVipul Gupta ([vipulgupta2048](http://www.letmegooglethat.com/?q=vipulgupta2048)) posts occasional on his blog, [Mixster](http://www.mixstersite.wordpress.com/). Check him out there for more updates like these.\n","notHasFeatureImage":false,"slug":"my-first-contribution-to-openebs-\\#oss"},{"id":95,"title":"Using Kubernetes Custom Resources for Microservices IPC","author":"Ganesh Kumar","author_info":"Gopher, Open Source Contributor, Thinker, Health enthusiast","date":"11-05-2018","tags":["Kubernetes","Golang","OpenEBS","Microservices"],"excerpt":"This blog talks about why I used Custom Resources as a way for communication between different microservices (aka Kubernetes Pods).","content":"\nThis blog talks about why I used [Custom Resources](https://kubernetes.io/docs/concepts/api-extension/custom-resources) as a way for communication between different microservices (aka Kubernetes Pods).\n\nOpenEBS is a fully containerized storage solution running within Kubernetes. Infact [OpenEBS](https://openebs.io/docs) extends the Kubernetes cluster functionality to manage storage and stateful workloads.\n\nOpenEBS has an operator (or orchestration component) called [Maya](https://github.com/openebs/maya) (magic) that relays Volume management operations to several storage engines. OpenEBS already supports [Jiva](https://github.com/openebs/jiva) based storage engine. The purpose is to plugin a more independent [cStor](https://github.com/openebs/cstor)based storage engine (making use of zfs on userspace).\n\nI essentially have a user interacting with OpenEBS Maya using PV, PVC, StorageClasses etc. and OpenEBS Maya interacts with cStor Pods. I will be focusing on the design considerations for interactions between OpenEBS Maya and cStor Pods.\n\n![Architecture](/images/blog/architecture.png)\n\n_1 — Shows user creating StoragePoolClaim (SPC) CR, with details like the number and type of pools to be created. Let us consider, SPC specifies a cStor pool should be created._\n\n_2 — Maya has a custom controller that watches for the SPC CR and it will go ahead and create the cStor Pods with a cstor-pool container and a side-car cstor-pool-mgmt that has a CLI interface for creating pools and volumes. Side-car container (following the Kubernetes Ambassador Pattern), helps in translating the Pool and Volume operations triggered by Maya into the corresponding CLI commands._\n\n_3 — Depending on the user’s request for creating a pool or a volume, Maya will create CStorPool and CStorVolumeReplica CRs respectively. Note that, I could have had the cstor-pool-mgmt container expose an API service that Maya could have invoked. Instead, I decided to use CRs and I will explain why in the following sections._\n\n_4 — cstor-pool-mgmt sidecar application watches for CRs of CStorPool and CStorVolumeReplica and performs pool — volume operations._\n\n**One of the core design constraint while deciding on inter-pod communication between Maya and CStor Pod is that:** _When the user requests for a Volume, the cluster state may not be fully ready to satisfy all the criteria — for example, User requests for 3 replicas but there are only 2 nodes running. The request should be cached and a third replica has to be provisioned whenever a new node gets added to the cluster._\n\nDevelopers normally go with an approach to make a REST/gRPC call to the receiver and store in a database, running in separate pod/statefulset. But OpenEBS thinks beyond that.\n\nNow consider, that I had used the traditional approach of using a REST/gRPC method of interactions between Maya and CStor Pods, then Maya would have to implement/consider cases like:\n\n- Where to store the state of current request, as the request can’t be serviced immediately depending on the cluster state. This is required to handle the case where the node running Maya can itself go down.\n- When working in a scaled environment, when there are multiple Maya pods, who gets to service the requests and when one of the pod goes down, should the other take it over or not?\n- How to handle the case where, Maya sends the request to the CStor container and then it goes boom (after all this is Kubernetes Cluster and they are supposed to handle all kinds of Chaos), who handles the results of the operation at CStor. In other words, how to implement a 2-phase commit?\n\n_However, if you look at my design constraint, doesn’t it sound similar to how a Kubernetes deployment with 3 replicas work?_ The user defines a desired state (in this case Maya) and the controllers make it happen eventually. So, why not just, be a roman when in rome.\n\n### Thats exactly what OpenEBS does!\n\nOpenEBS goes with `watch`-er approach. i.e., watch [**k8s custom resource definition**](https://kubernetes.io/docs/concepts/api-extension/custom-resources). If a pool (virtual disk) must be created on top of the actual disk, Maya creates a custom resource (named CStorPool), and a pod running cstor-pool-mgmt watcher gets an event for corresponding resource request and starts performing pool related operations. Cool, isn’t it?\n\n### Where does the custom resource get stored?\n\nkubernetes etcd. You pronounced it right, that’s `yetsed`, :-)\n\n### How is custom resource efficient?\n\n- Storing critical details in a separate pod-database, leads to pod level consistency. Storing in etcd leads to **cluster level consistency**.\n- Even if the **receiver is not running**, when the request is generated, the receiver-watcher gets an event, as and when it starts running.\n- Users can access the resources via **k8s cli** — `kubectl get `\n- Update the status of request on the same custom resource.\n\n### What is the problem with custom resource?\n\n- It is not suitable for transactional communication. (Say if an OTP request needs to be done within 20 seconds, it is not applicable to go with, “as and when up” approach).\n Solution: Before making any transactional call, verify status of receiver and make a REST/gRPC API call. No other go, :-(\n- It is slightly complex to implement watcher.\n Solution: My next blog will address an easy way to implement CRD watcher, how to solve issues with watcher design and different ways to implement watching controller. Practise, you become perfect; Follow us, you become fantastic :-)\n\nSupport and follow us [**@gkGaneshR**](https://twitter.com/gkGaneshR) and [**@openebs**](https://twitter.com/openebs) to get instant updates.\n\nThanks [Kiran](https://twitter.com/kiranmova) for your valuable support. We at OpenEBS are always looking for help and feedback from Community. Please join us on [Slack](https://slack.openebs.io/) or comment on the [design doc](https://docs.google.com/document/d/1Q5W3uHktHa-vOm8oGp-3kpAQ3V1tvyk5AYmxxtf57Rg/edit?usp=sharing) and related [Pull Request](https://github.com/openebs/maya/pull/284).\n\n**Summary**\n\n- K8s CRD becomes a good alternative to REST/gRPC API for “push to perform” operations.\n- Few more implementation details will be covered in upcoming blogs — Follow us for updates.\n","notHasFeatureImage":true,"slug":"using-kubernetes-custom-resources-for-microservices-ipc"},{"id":96,"title":"Litmus - Release a chaos monkey on your Kubernetes Stateful Workloads!","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"01-05-2018","tags":["Chaos","Kubernetes","Litmus","Software Testing","E2e","Chaos Engineering"],"excerpt":"If you are a Kubernetes Enthusiast and working on stateful workloads, you may be asking yourself","content":"\n**In this blog we quickly talk about what led us to build Litmus and to open source it.**\n\nIf you are a Kubernetes Enthusiast and working on stateful workloads, you may be asking yourself:\n\n“With all the options I have to run Kubernetes — the permutations are endless — how can I be sure that my particular mix of options works well end to end at keeping my data safe and accessible?”\n\nYou are not alone, as can be seen by the ever increasing conversations on Kubernetes sig-storage slack channel and other forums like Reddit or Twitter. To just pick a few conversations:\n\n![Is it really recommended to run stateful workloads like MySQL on Kubernetes?](https://cdn-images-1.medium.com/max/800/1*6VJXdgFpuwD-fUkEKPo0GA.png)\n\n[Is it really recommended to run stateful workloads like MySQL on Kubernetes?](https://www.reddit.com/r/kubernetes/comments/88fxdg/is_it_really_not_recommended_to_run_stateful/)\n\n![What are the storage solutions offered in Kubernetes today? Which one will suit my workload](https://cdn-images-1.medium.com/max/800/1*5s60fO7nzhZfC3SFNiY0gA.png)\n\n[What are the storage solutions offered in Kubernetes today? Which one will suit my workload](https://twitter.com/rothgar/status/978694465975083009)\n\nAnd say, you somehow have made the journey to explore different solutions out there, mostly referring to product documentation and blogs, how can you be sure that the solution will continue to work in your enterprise environment?\n\nAs enterprises move to DevOps and microservices, more and more of the infrastructure from policy engines through storage and everything in between such as DNS, tracing, logging and more are selected and operated by all in one teams. With this control and autonomy comes greater agility — and all too often, _stress_.\n\nMeanwhile, infrastructure vendors and projects are also (we know first hand) challenged to keep their end-to-end (e2e) and chaos engineering frameworks updated with the ever-increasing permutations of deployment scenarios. Kubernetes itself is changing, new providers emerge every day, workloads are changing, and all of it is increasingly simple to adopt and deploy. As a storage solution provider we simply cannot have the resulting explosion of “corner cases” go untested.\n\nThe solution providers can go one step ahead to open source their project, but it still doesn’t help the users to ensure that the selected Kubernetes stack works in their highly distributed and agile environments and they are not called to fight fires at 3 AM.\n\n![Fire-fighting production issues !!](https://cdn-images-1.medium.com/max/800/0*qX8CliW_E3gKMURn.)\n\n“What’s a person to do? Test, test, release the chaos monkeys, and test again!”\n\nThankfully, Kubernetes and containerization and Go and some software engineering we’re happy to share make it much easier to provide end to end validation in real world conditions !\n\n#### So — What is Litmus?\n\n**_Litmus is a community for e-2-e testing and chaos engineering for Kubernetes, focusing on stateful workloads._**\n\nThe primary objective of Litmus is to ensure a consistent and reliable behavior of Kubernetes for various persistent workloads and to catch hard-to-test bugs and unacceptable behaviors before users do. Litmus can detect many more real-world issues than relatively simple issues identified by unit and integration tests.\n\nLitmus can also be used to determine if a given Kubernetes deployment is suitable for stateful workloads. While Litmus tests and metrics were developed initially to test the resilience of container attached storage from OpenEBS and others — we realized that the use cases are broader and overall system resilience can be characterized, which is a major reason we are open sourcing our efforts and putting the time into starting the Litmus community.\n\nLitmus tests range from initial setup and configuration validation to deploying and running persistent workloads under various conditions and failures.\n\n_What sets Litmus apart is not just its intent of being an end to end testing framework that can be embedded into any CI/CD pipeline, but the ease with which different teams from product developers to customers can contribute to the tests. Litmus allows for defining scenarios using native language specifications (English !!) OR a set of easy-to-define/understand YAML templates which are internally converted into test scripts, with a simple Kubernetes manifest as the end-product._\n\nHere is a simple test, defined in plain English:\n\n![Simple test in plain english](https://cdn-images-1.medium.com/max/800/0*ar6cYX2rEJ7Nh_G2.)\n\n## How to get involved with Litmus?\n\nFirst, it might be useful to understand the basic pieces of Litmus. Litmus has the following major components:\n\n![Litmus: High level architecture](https://cdn-images-1.medium.com/max/800/1*CdBbpkSilx3aJnZA3tiAjQ.png)\n\n- **Deployments** that help in setting up different types of Kubernetes Clusters like on-premise, cloud, OpenShift, etc. The default is that the deployments provision and configure OpenEBS storage, however, these deployments are easily extended to support other storage and we are happy to help any user or storage vendor to build additional deployments.\n- **Facilitators** for test execution that aid: defining and running test suites, capturing logs and generating reports about the test runs, fault/error injection tools that help to perform chaos tests, examples that demonstrate how to integrate these test pipelines with Slack notifications\n- **Test modules** that are triggered from within a Kubernetes cluster. Think of these as containerized tests. For instance, the **_mysql-client_** can be launched as a pod to validate MySQL resiliency while the underlying nodes and the connected storage are subjected to chaos engineering.\n- **Tests** that themselves are written in easy to understand formats, either in plain English (thanks [Godog](https://github.com/DATA-DOG/godog)!) or in Ansible Playbooks. These tests primarily interact with the Kubernetes cluster via **_kubectl_** making them highly portable.\n\nLitmus can be used to test a given workload in a variety of Kubernetes environments, for example, a developer minikube or a GKE cluster with a specific storage solution or as a part of a full-fledged CI setup.\n\nLitmus is early and needs all the help you can provide to have it cover the ever-growing Kubernetes landscape. Checkout the [Litmus Project](https://github.com/openebs/litmus) on Github for more details or if you are at KubeCon EU, please join us for the talk this Friday on [End to End testing with Kubectl](https://kccnceu18.sched.com/event/DqwD/using-kubectl-to-run-your-end-to-end-tests-amit-kumar-das-uday-kiran-mayadata-intermediate-skill-level) to learn more about how we have built Litmus and a quic\n\n#### Conclusion:\n\nPlease welcome Litmus into the world! We’re pretty sure it addresses a set of needs being felt by everyone from developers and operators to service providers and cloud native open source projects such as OpenEBS. With Litmus we use microservices and containers and Kubernetes to test, validate and characterize environments end to end. Your feedback is welcome and needed. Thanks for reading!\n","notHasFeatureImage":false,"slug":"litmus---release-a-chaos-monkey-on-your-kubernetes-stateful-workloads!"},{"id":97,"title":"How do I create an OpenEBS storage pool on Google Persistent Disk","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"13-04-2018","tags":["Docker","OpenEBS","Solutions","Google Cloud Platform","Kubernetes"],"excerpt":"The OpenEBS volume replicas, which are the actual backend storage units of the OpenEBS iSCSI target currently store the data in a hostPath on the Kubernetes nodes.","content":"\nThis article belongs to #HowDoI series on Kubernetes and OpenEBS.\n\nThe OpenEBS volume replicas, which are the actual backend storage units of the OpenEBS iSCSI target currently store the data in a hostPath on the Kubernetes nodes. By default, a folder with the volume (PV) name is created on the root filesystem, in a parent directory (/var/openebs) & bind mounted into the container during the replica pod instantiation. This parent directory (also created if not already available), which is basically a persistent path to hold the individual volumes is referred to as a **_Storage Pool_**.\n\nNote: The notion of the storage pool described above is specific to the current default storage engine,i.e., Jiva. Future releases may see availability of additional storage-engines which can consume block devices instead of hostdir to create storage pools\n\nFor various reasons, it may be desirable to create this storage pool on an external disk (GPD, EBS, SAN) mounted into specific locations on the Kubernetes nodes. This is facilitated by the **OpenEBS storage pool policy**, which defines the storage pool as a **_Kubernetes Custom Resource_** with the persistent path as an attribute.\n\nThis blog will focus on the steps to be followed to create the OpenEBS PV on Google Persistent Disks (GPD).\n\n## PRE-REQUISITES\n\n- 3-Node GKE cluster with the OpenEBS Operator installed.\n- 3-Google Persistent Disks, one attached to each node of the cluster.This can be done using the **_gcloud compute disks create_** & **_gcloud compute instances attach-disk_** commands (Refer for console steps: [https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk](https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk))\n\n### STEP-1: Format the GPDs & Mount into desired path\n\nOn each node, perform the following actions :\n\n- Switch to root user _sudo su –_\n- Identify GPD attached _fdisk -l_\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# fdisk -l\n Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors\n Units: sectors of 1 \\* 512 = 512 bytes\n Sector size (logical/physical): 512 bytes / 4096 bytes\n I/O size (minimum/optimal): 4096 bytes / 4096 bytes\n Disklabel type: dos\n Disk identifier: 0x635eaac1\n\n Device Boot Start End Sectors Size Id Type\n /dev/sda1 \\* 2048 209715166 209713119 100G 83 Linux\n\n Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors\n Units: sectors of 1 \\* 512 = 512 bytes\n Sector size (logical/physical): 512 bytes / 4096 bytes\n I/O size (minimum/optimal): 4096 bytes / 4096 bytes\n ```\n\n- Format the disk with, say ext4 fs (_mkfs.ext4 /dev/sd<>)_\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mkfs.ext4 /dev/sdb\n mke2fs 1.42.13 (17-May-2015)\n /dev/sdb contains a ext4 file system\n last mounted on /openebs on Fri Apr 13 05:03:42 2018\n Proceed anyway? (y,n) y\n Discarding device blocks: done\n Creating filesystem with 2621440 4k blocks and 655360 inodes\n Filesystem UUID: 87d36681-d5f3-4169-b7fc-1f2f95bd527e\n Superblock backups stored on blocks:\n 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632\n\n Allocating group tables: done\n Writing inode tables: done\n Creating journal (32768 blocks): done\n Writing superblocks and filesystem accounting information: done\n ```\n\n- Mount the disk into desired mount point (_mount -o sync /dev/sd<> /mnt/openebs_)\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mount -o sync /dev/sdb /mnt/openebs/\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mount | grep openebs\n /dev/sdb on /mnt/openebs type ext4 (rw,relatime,sync,data=ordered)\n ```\n\n### STEP-2 : Create a storage pool custom resource\n\n- Construct a storage pool resource specification as shown below & apply it (Note that the custom resource definition for the storage pool is already applied as part of the operator install)\n\n ```\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: sp-mntdir\n type: hostdir\n spec:\n path: \"/mnt/openebs\"\n ```\n\n### STEP-3 : Refer the storage pool in a custom storage class\n\n```\n---\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n name: openebs-custom\nprovisioner: openebs.io/provisioner-iscsi\nparameters:\n openebs.io/storage-pool: \"sp-mntdir\"\n openebs.io/jiva-replica-count: \"3\"\n openebs.io/volume-monitor: \"true\"\n openebs.io/capacity: 5G\n```\n\n### STEP-4 : Use the custom storage class in an application’s PVC spec\n\n```\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: demo-vol1-claim\nspec:\n storageClassName: openebs-custom\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n```\n\n### STEP-5 : Confirm volume is created on the storage pool\n\n- Once the OpenEBS PV is created (_kubectl get pv, kubectl get pods_), list the contents of the custom persistent path mentioned in the storage pool custom resource. It should contain a folder with the PV name consisting of the sparse files (disk image files)\n\n ```\n karthik_s@strong-eon-153112:~$ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-556e7ab7-3ed9-11e8-8e6a-42010a800216 5G RWO Delete Bound default/demo-vol1-claim openebs-custom 59m\n\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# ls /mnt/openebs/\n lost+found pvc-556e7ab7-3ed9-11e8-8e6a-42010a800216\n ```\n\n### GOTCHAS !!\n\n_Issue_: GPDs are detached in the event of a) Cluster resize (downscale/upscale) , b) upgrades & c) VM halts\n\n- No options to add “additional disks” during cluster creation\n- Instance templates are “immutable”, disks have to be added to instances separately\n\n_Workaround_: Perform a manual re-attach in above situations (Enlarged root disks are an option, but generally not recommended)\n","notHasFeatureImage":false,"slug":"how-do-i-create-an-openebs-storage-pool-on-google-persistent-disk"},{"id":98,"title":"Using chaoskube with OpenEBS.","author":"Sudarshan Darga","author_info":"Senior Software Engineer at MayaData","date":"12-04-2018","tags":["OpenEBS","Chaos Engineering","Kubernetes","Solutions"],"excerpt":"Chaos Engineering is the discipline of proving the reliability of any system by causing “chaos”. The word ‘Chaos’ means the state of confusion or failure caused due to unexpected reason.","content":"\n**Chaos Engineering** is the discipline of proving the reliability of any system by causing “chaos”. The word ‘Chaos’ means the state of confusion or failure caused due to unexpected reason.\n\n### Failures can be caused due to:\n\n- Power outages.\n- Software bugs.\n- Human Error.\n\n### Since failure is unavoidable.\n\n- Why not deliberately introduce failure to ensure the system can deal with the failure?\n- Chaoskube is one such tool, which can be used to introduce pod failures on Kubernetes Cluster.\n\n### Overview of Chaoskube:\n\n- Chaoskube is an open source Chaos Testing tool.\n- Written in GO language.\n- Can induce pod/controller failures on K8s Cluster.\n- Can kill pods by specifying the labels, namespaces.\n- Simple and easy to run.\n\n## Setup Chaoskube infrastructure on 3 node Kubernetes Cluster:\n\n```\nubuntu@kubemaster-01:~$ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\nkubemaster-01 Ready master 2d v1.8.8\nkubeminion-01 Ready 2d v1.8.8\nkubeminion-02 Ready 2d v1.8.8\nkubeminion-03 Ready 2d v1.8.8\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/e2e/ansible/playbooks/resiliency/test-ctrl-failure/chaoskube.yaml\ndeployment \"chaoskube\" created\nserviceaccount \"chaoskube\" created\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/e2e/ansible/playbooks/resiliency/test-ctrl-failure/rbac.yaml\nclusterrole \"chaoskube\" created\nclusterrolebinding \"chaoskube\" created\nubuntu@kubemaster-01:~$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\nchaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 32s\nmaya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\nopenebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\n```\n\n## Deploy Percona application on OpenEBS volume with Liveness probe:\n\n```\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/elves/master/e2e/percona-liveness/percona.yaml\ndeployment \"percona\" created\npersistentvolumeclaim \"demo-vol1-claim\" created\nservice \"percona-mysql\" created\nubuntu@kubemaster-01:~$ kubectl create configmap sqltest https://raw.githubusercontent.com/openebs/elves/master/e2e/percona-liveness/sql-test.sh\nconfigmap \"sqltest\" created\nubuntu@kubemaster-01:~$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\nchaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 6m\nmaya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\nopenebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\npercona-85b8997987-dg6jm 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-vd8t5 2/2 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-64rv5 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-b5v25 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-gs69w 1/1 Running 0 1m\n```\n\n## Induce controller failure using Chaoskube:\n\n- Induce failure on pod with label ‘openebs/controller=jiva-controller’ for duration of 60 seconds with interval of 20 seconds, which means it will induce controller pod failure for every 20 seconds for 3 times.\n\n ```\n kubectl exec chaoskube-55fc8f5f6d-tb6hj -- timeout -t 60 chaoskube --labels 'openebs/controller=jiva-controller' --no-dry-run --interval=20s --debug\n ```\n\n- Output should look something like this:\n\n ```\n time=\"2018-04-13T09:50:36Z\" level=info msg=\"terminating pod\" name=pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-vd8t5 namespace=default\n time=\"2018-04-13T09:50:36Z\" level=debug msg=sleeping duration=20s\n ```\n\n- Lets observe the failure induces by watching `kubectl get pods` for every 2 seconds.\n\n ```\n Every 2.0s: kubectl get pods Fri Apr 13 09:55:35 2018\n NAME READY STATUS RESTARTS AGE\n chaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 16m\n maya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\n openebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\n percona-85b8997987-dg6jm 1/1 Running 7 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-nh5fk 2/2 Running 0 15s\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-64rv5 1/1 Running 0 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-b5v25 1/1 Running 0 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-gs69w 1/1 Running 0 12m\n ```\n\n- Observe that percona application pod with liveness probe is still running after inducing openebs controller pod failure using chaoskube. Hence, the system is reliable after causing ‘Chaos’.\n\n### Reference Links:\n\n- [https://github.com/linki/chaoskube](https://github.com/linki/chaoskube)\n- [https://openebs.io/docs](/docs)\n","notHasFeatureImage":false,"slug":"using-chaoskube-with-openebs."},{"id":99,"title":"How do I pin the OpenEBS Replica Pod(s) to the Kubernetes Nodes where they were scheduled?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"26-03-2018","tags":["Howdoi","Kubernetes","OpenEBS","Storage","Solutions","Tutorials"],"excerpt":"A OpenEBS Volume comprises of a Controller pod and one or more Replica pod(s). Controller pod (also known as a Target pod) is the one to which the application can make an iSCSI connection.","content":"\nThis article belongs to #HowDoI series on Kubernetes and OpenEBS.\n\nA OpenEBS Volume comprises of a Controller pod and one or more Replica pod(s). Controller pod (also known as a Target pod) is the one to which the application can make an iSCSI connection. The Replica pods are the ones that access the underlying disk resources for storing the data.\n\n**Use Case #1: In my Kubernetes cluster, [_OpenEBS volume pods are scheduled on appropriate nodes_](/blog/how-do-i-configure-openebs-to-use-storage-on-specific-kubernetes-nodes). This is all fine till the cluster experiences a disruption due to network partition. Kubernetes tries to evict & re-schedule these volume pods into newer nodes that does not have the underlying data. This results in volume getting into offline state. I want the OpenEBS volume pods to stick to the nodes they were originally placed.**\n\n**Solution**: Patch the Replica deployment with **nodeAffinity** property\n\nAs per Kubernetes docs, nodeAffinity allows you to constrain which nodes your pod is eligible to be scheduled on. It is based on labels on the node.\n\nThere are currently two types of node affinity:\n\n– `requiredDuringSchedulingIgnoredDuringExecution` &\n\n– `preferredDuringSchedulingIgnoredDuringExecution`\n\nThese node affinity types can be thought of “_hard_” vs. “_soft_” affinity respectively.\n\n- _Hard_ affinity states that pod will be scheduled only if the conditions are met.\n- _Soft_ affinity implies Kubernetes will make a best effort but the affinity may not be guaranteed.\n\nWe shall make use of `hard affinity` as this fits perfectly to the needs of Replica deployment.\n\nSteps required to patch the Replica deployment are summarised below:\n\n**Step 1**:- Replica pod(s) gets scheduled by Kubernetes default scheduler (via OpenEBS provisioner — a dynamic Kubernetes storage provisioner)\n\n**Step 2**:- Wait till replica pod(s) get into `Running` state\n\n**Step 3**:- Operator determines the node(s) on which the replica pod(s) are scheduled\n\n**Step 4**:- Replica deployment is patched with nodeAffinity\n\n```bash\n# REPLACE WITH ACTUAL NAMESPACE\n# REPLACE WITH ACTUAL PV NAME\n# TAKE A NOTE OF THE NODE NAME(S) TO BE USED IN THE PATCH.YAML\n\nkubectl get po -n \\\n -o=custom-columns=NAME:metadata.name,NODE:spec.nodeName,STATUS:status.phase \\\n | grep -E 'NAME|-rep'\n```\n\n```bash\n$ cat replica_patch.yaml\n```\n\n```yaml\nspec:\n template:\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - nodename_where_replica_pod_1_got_scheduled\n - nodename_where_replica_pod_2_got_scheduled\n - nodename_where_replica_pod_3_got_scheduled\n```\n\n```bash\n# REPLACE WITH ACTUAL PV NAME\n\nkubectl patch deployment -rep \\\n -p \"$(cat replica_patch.yaml)\"\n```\n\n```bash\n# VERIFY IF PODs ARE BACK TO `Running` AFTER PATCH\n# REPLACE WITH ACTUAL NAMESPACE\n# REPLACE WITH ACTUAL PV NAME\n\nkubectl get po -n \\\n | grep -E 'NAME|-rep'\n```\n\nLearn more about nodeAffinity from Kubernetes docs at [https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity).\n\nIf you want to understand more on kubectl patch operation, then go through [https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/](https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/).\n","notHasFeatureImage":true,"slug":"how-do-i-pin-the-openebs-replica-pod(s)-to-the-kubernetes-nodes-where-they-were-scheduled?"},{"id":100,"title":"How do I configure OpenEBS to use storage on specific Kubernetes nodes?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"18-03-2018","tags":["Howdoi","OpenEBS","Solutions","Kubernetes","Tutorials"],"excerpt":"A OpenEBS Volume comprises of a Target pod and Replica pod(s). There can be one or more Replica pods. The Replica pods are the ones that access the underlying disk resources for storing the data.","content":"\nThis article belongs to **#HowDoI** series on [**Kubernetes**](https://kubernetes.io/) and [**OpenEBS**](https://openebs.io/).\n\n**Note: The approach mentioned in this article applies for OpenEBS version 6.0 or below. One can refer to this [link](https://github.com/openebs/community/pull/20) for OpenEBS version 0.8.0 and above.**\n\nA OpenEBS Volume comprises of a Target pod and Replica pod(s). There can be one or more Replica pods. The Replica pods are the ones that access the underlying disk resources for storing the data.\n\n**Use Case #1: In my Kubernetes Cluster, I have certain nodes that have disks attached. I call these as Storage Nodes. I want the OpenEBS Volume Replica Pods to be scheduled on these Storage Nodes.**\n\n**_Solution:_** Use Kubernetes “taints & tolerations” feature.\n\nAs per Kubernetes docs, taints allow a node to repel a set of pods. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes.\n\n- You can apply `NoSchedule` & `NoExecute` taints to the node(s).\n- `NoSchedule` marks that the node should not schedule any pods that do not tolerate the taint.\n- `NoExecute` marks that the node should evict existing/running pods that do not tolerate this taint.\n- Tolerations are applied to pods, and allow the pods to get scheduled onto nodes with matching taints.\n- You need to set an ENV variable in maya API server Deployment specifications, which in turn ensures setting of above tolerations on the replica pods.\n- The ENV variable referred to here is `DEFAULT_REPLICA_NODE_TAINT_TOLERATION`\n\nFollowing are the instructions to do the same:\n\n````\n# Step 1  —  Taint the node(s)\n```bash\n# kubeminion-01 is the name of a Kubernetes node\n# The taint effects used here are `NoSchedule` and `NoExecute`\nkubectl taint nodes kubeminion-01 storage=ssd:NoSchedule storage=ssd:NoExecute\n```\n\n# Step 2  —  Maya API server should be deployed with below specs\n# This ensures the replica pods are set with appropriate tolerations\n```yaml\napiVersion: apps/v1beta1\nkind: Deployment\nmetadata:\n name: maya-apiserver\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n name: maya-apiserver\n spec:\n serviceAccountName: openebs-maya-operator\n containers:\n —  name: maya-apiserver\n imagePullPolicy: Always\n image: openebs/m-apiserver:0.5.3\n ports:\n —  containerPort: 5656\n env:\n —  name: DEFAULT_REPLICA_NODE_TAINT_TOLERATION\n value: storage=ssd:NoSchedule,storage=ssd:NoExecute\n```\n````\n\n**Use Case #2:** In my Kubernetes Cluster, I have certain nodes that have disks attached. I call these as Storage Nodes. I want the OpenEBS Volume Replica Pods to be scheduled on these Storage Nodes. In addition, I want a better utilization of these nodes by being able to schedule my application Pods on these nodes as well.\n\n**Solution:** Use Kubernetes taints & tolerations feature. You may also want to try with `nodeAffinity` to achieve this. However, this solution focuses on use of tolerations.\n\n- You need to make use of `PreferNoSchedule` as the taint effect.\n- This can be thought of as a _soft version_ of `NoSchedule`.\n- In other words the system tries to avoid placing a pod that does not tolerate the taint on the node, but it is not mandatory.\n\nFollowing are the instructions to do the same:\n\n````\n# Step 1  —  Taint the node(s)\n```bash\n# kubeminion-01 is the name of a Kubernetes node\n# The taint effect used here is `PreferNoSchedule` i.e. a soft version of `NoSchedule` \n# the system tries to avoid placing a pod that does not tolerate the taint on the node,\n# but it is not mandatory.\nkubectl taint nodes kubeminion-01 storage=ssd:PreferNoSchedule\n```\n\n# Step 2  —  Maya API server should be deployed with below specs\n# This ensures the replica pods are set with appropriate tolerations\n```yaml\napiVersion: apps/v1beta1\nkind: Deployment\nmetadata:\n name: maya-apiserver\n namespace: default\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n name: maya-apiserver\n spec:\n serviceAccountName: openebs-maya-operator\n containers:\n —  name: maya-apiserver\n imagePullPolicy: Always\n image: openebs/m-apiserver:0.5.3\n ports:\n —  containerPort: 5656\n env:\n — name: DEFAULT_REPLICA_NODE_TAINT_TOLERATION\n value: storage=ssd:PreferNoSchedule\n ```\n````\n\nIf you want to learn more on taints & tolerations, then go through [https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)\n\nI shall strive to put more such articles in future. Do let me know if you want any specific topics that I should explain.\n\n_Thanks to Kiran Mova._\n","notHasFeatureImage":true,"slug":"how-do-i-configure-openebs-to-use-storage-on-specific-kubernetes-nodes?"},{"id":101,"title":"Achieving native hyper convergence in Kubernetes","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"13-03-2018","tags":["Hyper Convergence","Kubernetes","Nutanix","OpenEBS","Persistent Storage"],"excerpt":"Hyper convergence has a lot of benefits — which is one reason it has become popular in the traditional infrastructure-centric world of virtual machines with proprietary vendors like Nutanix reaching prominence over the last several years.","content":"\nHyper convergence has a lot of benefits — which is one reason it has become popular in the traditional infrastructure-centric world of virtual machines with proprietary vendors like Nutanix reaching prominence over the last several years.\n\nKubernetes is nearly ready as a layer enabling hyper convergence, as the compute orchestration is extremely flexible and networking has moved to a largely containerized approach that leverages local resources in attached physical and virtual hosts.\n\nWhen it comes to storage, however, there are a few pieces that are missing. Once added to Kubernetes, these pieces will unlock a number of benefits to users of Kubernetes including better resource utilization, reduction of noisy neighbor phenomena, simpler management, isolation at the node level thereby reducing the potential blast radius of failures, and, perhaps most importantly, further ownership and management of relevant infrastructure per workload and per DevOps team.\n\nStorage management capabilities in Kubernetes have improved in the last couple of years. For example, there is now clarity around how to connect a stateful application to persistent storage. The constructs of persistent volume claim (PVC), persistent volume (PV), and storage class (SC) along with dynamic provisioners from vendors have clarified how to connect a pod to a storage volume. With these Kubernetes constructs, a large ecosystem of legacy storage found its way to be connected to application pods. Many vendors and open source projects are so excited about this connectivity to cloud native environments that they have taken to calling their traditional storage “[cloud native](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing)”.\n\nIn order to explain why new tools and constructs are needed to improve the management of storage media, let’s start by reviewing pod connectivity. Shown below is a pod connected to external storage through a dynamic provisioner interface.\n\n![Need for new tools and constructs in Kubernetes for managing disks](https://cdn-images-1.medium.com/max/800/1*zm4UFgEvTWesM2JoxQF9Cg.jpeg)\n\nIn addition, we show the Local PV construct connected to local disks whether spinning or solid state.\n\nCurrently the Local PV can manage just a single disk. In a typical hyper-converged solution, more disks would be involved for a given pod. In addition to this single disk limitation, the following are limitations or gaps in the local PV feature.\n\n- Add disks / detach disks to a pod currently requires the pod to be restarted\n- Control and access to the storage is itself is limited. There will be a number of benefits of enabling access at times to the level of NUMA and to allowing CPU cores to be attached to the storage pod; keep in mind that these days using all the cores itself can be a challenge and driving up utilization is one of the central attributes of hyper converged systems\n- Today there is no standard ability to share the underlying disk. This is particularly important as extremely fast, and relatively expensive, NVMe SSD devices are now readily available and if shared they could be used as a cache for multiple Persistent Volumes\n- Lack of any fault management capabilities. Kubernetes needs to be able to receive and manage storage faults such as failures of the underlying disk, changes to the latency of disk IO\n\nWith these limitations in mind, we summarize needed enhancements as:\n\n- Capability to pool the disks, and provide an interface to manage the pooling\n- Capability to monitor disks, identify faults, forward them to the appropriate receivers\n\nIn the drawing above, these requirements are loosely shown as “constructs and tools to manage local disks”. These disk related constructs and tools are largely meant to be managed by Kubernetes cluster administrators and DevOps administrators in a Kubernetes like way.\n\nIt may be easier to think about this using human personas. Let’s say a DevOps developer is interested in connecting a working storage volume to their workload. Meanwhile a DevOps admin wants to rely as much as possible on Kubernetes to deliver storage services.\n\nToday, DevOps admins are forced to turning to different storage solutions to create storage classes as opposed to having a generic way of writing solutions around creating storage classes in a k8s native way. The DevOps admin would love to have a native k8s way to create storage classes so that they can standardize on an approach irrespective of the underlying storage systems or even storage cloud services.\n\n![Using and constructing a storage class in Kubernetes](https://cdn-images-1.medium.com/max/800/1*17YT5-GR_JUXEq6qW2SD1A.jpeg)\n\n## Some thoughts on what these new disk related constructs and tools could be\n\nJust like storage connectivity issue is solved with a dynamic volume provisioner, we could introduce pool provisioners into Kubernetes.\n\n![Proposed disk related constructs and interfaces](https://cdn-images-1.medium.com/max/800/0*eM2LjKDvhbl62mjG.)\n\nAs shown above, DevOps administrators will have the required tools to design the storage policy decisions. A toolset called is created node-disk-manager to provision, monitor, and manage disks on the node. The disks are then grouped into pools by an interface called pool provisioner. The pool provisioner gives a generic set of APIs to consume the Kubernetes disk objects and create a storage technology specific pools such as OpenEBS cStorPool, OpenZFS zpool, GlusterPool etc.The advantage of representing the pools in native Kubernetes constructs is that Kubernetes native tools can be extended to manage these new constructs.\n\nWith these constructs, the end-to-end volume provisioning work flow could be depicted like below.\n\n![New proposed workflow for managing local disks and achieving true hyper convergence in Kubernetes](https://cdn-images-1.medium.com/max/800/1*9bAs7wOPNNGLxELpgP-4FA.jpeg)\n\n## Conclusion:\n\nKubernetes native constructs and tools to manage and monitor local disks will help move towards achieving true hyper-convergence. We have observed many users in the OpenEBS community are using OpenEBS dynamic provisioner seamlessly and requesting tools to manage the disks and storage pools. We are thinking of adding such tools to Kubernetes itself so that they are available to the larger community. A draft design proposal is available at We are looking forward to proposing and discussing these thoughts with K8S SIG leadership.\n\nWe are cooking up some draft proposals in the OpenEBS project before we take the to K8S SIG. Your feedback is appreciated and needed.\n\n### Disk Manager design:\n\n[https://docs.google.com/presentation/d/11GLg21x7G-nMTNw8aNIOhhjW\\_-eK19zSI9Xm-0jYHKs/edit?usp=sharing](https://docs.google.com/presentation/d/11GLg21x7G-nMTNw8aNIOhhjW_-eK19zSI9Xm-0jYHKs/edit?usp=sharing)\n\n[https://github.com/kmova/node-bot/blob/3cb83976a5392003d02275f8a94d1860257915f0/design/node-storage-management.md](https://github.com/kmova/node-bot/blob/3cb83976a5392003d02275f8a94d1860257915f0/design/node-storage-management.md)\n\n### Pool Provisioner design:\n\n[https://github.com/kmova/openebs/blob/35fb65540b17ad3da3df270ccc425c4ec417ca12/contribute/design/proposal-cstor-orchestration.md](https://github.com/kmova/openebs/blob/35fb65540b17ad3da3df270ccc425c4ec417ca12/contribute/design/proposal-cstor-orchestration.md)\n","notHasFeatureImage":true,"slug":"achieving-native-hyper-convergence-in-kubernetes"},{"id":102,"title":"The Mule and (the) Flash — going for a run?","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"26-02-2018","tags":["Kubernetes","Cloud Storage","Container"],"excerpt":"In this blog, I discuss why we are building an innovative approach to user IO for the purpose of containerized storage, in particular vhost. If you just want the code, take a look at https://github.com/openebs/vhost-user.","content":"\nIn this blog, I discuss why we are building an innovative approach to user IO for the purpose of containerized storage, in particular vhost. If you just want the code, take a look at [https://github.com/openebs/vhost-user](https://github.com/openebs/vhost-user).\n\nFirst off, as mentioned in previous blog articles, OpenEBS is not yet another distributed file system. Let’s reiterate the reasoning behind this:\n\n- As microservices typically require only a small (relatively) amount of storage, there is no need to build a scale-out storage system\n- As Direct-attached Storage (DAS), in particular, NVMe, is the fastest storage you can get, you want the workload and the controller to be local with respect to each other; this is true even with SSD cloud storage offerings like AWS EBS instances\n- As single NVMe devices can reach 450K IOPS per device or [more](https://www.prnewswire.com/news-releases/supermicro-delivers-groundbreaking-18-million-iops-of-storage-performance-in-new-2u-ultra-server-300508258.html) there is no longer any need to “scale out” to achieve high IOPS or low latency, in fact, scale-out adds latency as per the above argument\n\nFinally, distributed applications are complex by nature. When you are building microservices, you are in fact, developing a distributed application. It seems unwise to put one distributed application on top of the other (storage) and sleep well at night. All that work you’ve done limiting single points of failure in your application layer can be undone through the use of complex distributed storage.\n\nAnother fundamental aspect of OpenEBS is that it runs in user space. This too has, we like to believe, a significant advantage as it does not require you to [build a kernel module](https://github.com/portworx/px-fuse) and taint your kernel (in case of closed source) with out-of-tree code. But it does not stop there; if you want to move your data from cloud to cloud (c2c), you do not have to worry about kernel version mismatches or anything like that. User space is the new kernel — when it comes to IO.\n\nBut what about performance? Linus Torvals himself said some years ago that file systems in user space are nothing but [toys](https://www.phoronix.com/scan.php?page=news_item&px=OTYwMA). But, as it turns out, with these low latency SSDs and high-speed networking (100GbE) the kernel, in fact, has become the bottleneck!\n\n_“fuse works fine if the thing being exported is some random low-use interface to a fundamentally slow device.”_\n\nSo it seems that we have reached an impasse? The kernel appears to be the bottleneck, and user space implementations are just “toys.” Or have we? When you look into why IO in user space is slow, it’s mostly due to the inability to do DMA, the required context switches, and the copying in and out of data. What if we could avoid this? Also, as you may know, hardware is already causing a change in the way we do things — 3D XPoint™ next to NVMe. This can be seen by technologies applied in SPDK and others like FD.IO. As OpenEBS is storage in containers, we have started to work on what we call the IOC, or the IO Container using these technologies.\n\n**The IOC runs in user space and can do IO to the underlying hardware, bypassing the kernel altogether. It owns a set of resources (CPU, NICs, memory, and storage) and applies polling for IO instead of being interrupt driven.**\n\nWith 18-core desktop computers being available today, it’s hardly an issue to use a core or two dedicated for IO — in the user space.\n\nBecause the IOC exposes block devices, we need a way to connect these devices to the other containers. Luckily — the VM space solved that problem for us: v[host](http://www.spdk.io/doc/vhost.html). By reusing these approaches, we create high a speed connection between the IOC and the containerized storage controller without making a change to the applications.\n\n![IOC model with vhost user interface and VPP](/images/blog/ioc-model-with-vhost-user-interface-and-vpp.png)\n\nThe above picture tries to depict the situation on a single node. As the application sends its IO through a block protocol (the target), OpenEBS — through the shared vhost subsystem — sends the IO to the replica which applies storage logic to it. With storage logic, we mean things that allow OpenEBS to do Copy-on-write (COW), snapshots, clones, compression or whatever is required. Also, OpenEBS is starting to further leverage this architecture to alter data management parameters including replication and snapshot patterns and even lower level parameters as well as block size in those containers depending on the workload.\n\nThen finally, the IO is submitted again to the IOC where an adaptive polling algorithm waits for its completion. Note, that the target — replicates `n` copies to the other node(s) which is depicted with R(n). So instead of doing IO through the kernel, your application passes the IO to the IOC which takes care of completing the IO as fast as possible all from user space.\n\nWith this approach, we get the best of both worlds and are in fact capable of surpassing the performance you would get when doing the same in the kernel — hands down — while also providing per workload granularity of control.\n\n![User space outperforming the kernel](/images/blog/user-space-outperforming-the-kernel.png)\n\nAs you can see from the repository, the design is fairly straightforward and is intended to support both legacy workloads as well as those built for faster underlying storage. We welcome input and contributions from anyone.\n\nWhile the vHost work stands alone it is central to a new storage engine forthcoming in OpenEBS 0.6, code named ‘cStore’.\n\nWe would really like your input so please [open an issue](https://github.com/openebs/vhost-user/issues) or join us on Slack to discuss at [openebs-community.slack.com](http://openebs-community.slack.com/) or just contact me directly. I can be reached at Twitter at [@jeffrymolanus](https://twitter.com/jeffrymolanus)\n","notHasFeatureImage":false,"slug":"the-mule-and-(the)-flash-—-going-for-a-run?"},{"id":103,"title":"OpenEBS plus Red Hat OpenShift and StackPointCloud and IBM Cloud Private and….","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"05-02-2018","tags":["DevOps","Docker","Kubernetes","Updates","OpenEBS"],"excerpt":"This week we announced that our partnership with Red Hat is flourishing. We achieved their Primed level of certification for their OpenShift offerings and are seeing more and more users rely upon OpenShift and Kubernetes as a means to provide persistence to their workloads.","content":"\nThis week we [announced](https://www.prnewswire.com/news-releases/openebs-certified-with-red-hat-openshift-stackpointcloud-and-ibm-cloud-672729373.html) that our partnership with Red Hat is flourishing. We achieved their Primed level of certification for their OpenShift offerings and are seeing more and more users rely upon OpenShift and Kubernetes as a means to provide persistence to their workloads.\n\nThe alternative pattern, of course, is to connect to an external storage system. Solutions like Rook and others such as the CSI efforts of RexRay and many others enable the use of external storage.\n\nActually — so does OpenEBS :)\n\nOpenEBS can and often does use external storage underneath. With OpenEBS, however, every workload has its own storage controller(s) that themselves are easily orchestrated by Kubernetes and data is local by default. There are three main benefits to the OpenEBS containerized architecture that external only storage cannot address due to architectural limitations:\n\n- **The granularity of control** — with OpenEBS the storage controller interprets ever more individualized and extensive storage policies and makes them so for each workload. Because OpenEBS is a full system (or is becoming one :)), it offers far more control than centralized storage that itself has to address the sometimes competing needs of countless — hundreds — of workloads. Developer teams can take on storage knowing they are much less constrained than they are working with least common denominator external storage.\n- **No SPOF** — in an age in which chaos engineering is becoming more and more popular, the notion of a sacrosanct dependency that cannot itself be disrupted or the entire system crashes potentially into a non-recoverable state is anachronistic. Put more directly — shared scale-out storage is an anti-pattern for many. Blast Radius.\n- **Performance** — as storage heads, we too often likely focus on performance. However, OpenEBS does work with databases, and in some cases the speed at which you run those workloads directly translates into user experience and hence money. So the tax a scale out storage system puts on performance versus the insane and rapidly accelerating speed of direct attached is essential. Ironically, scale-out first arose in part to work around how slow local disk was; times have changed. If you are interested in performance, you’ll want to grab our cStore by the way which, as the name suggests, is written in C and does much else as well to build upon our inherently faster Container Attached Approach. Stay tuned…\n\nSo why Red Hat and why StackPointCloud?\n\nIn both cases, we see organizations that are doing an incredible job helping their target users adopt Kubernetes based orchestration. With Red Hat, we tend to see especially larger enterprises taking the approach. With StackPointCloud, there is a real mix of departmental level users at large organizations as well as countless start-ups. In both cases, our support of Helm charts for OpenEBS makes it trivial to spin OpenEBS up.\n\nWhile OpenEBS itself as not achieve 1.0 status, we are working hand in hand w/ partners to make sure users are succeeding in their use of OpenEBS for stateful workloads. There must now be at least hundreds of production proof of concept deployments ongoing.\n\nPlease get in touch via [Slack](https://join.slack.com/t/openebs-community/shared_invite/enQtMjQzMTg4NTcyNTY2LTJiMzVjYjA5ZDk3YmI4NjAxY2QyYmI3MTA1MmUxMTAzNTU0NTM5NTViOTIxMjA1NWQ4NzVmMTBiNjk0NDU1YzQ) or otherwise if you would like to spend a little time with us to discuss your use cases and, of course, if you are running OpenEBS and testing it out.\n","notHasFeatureImage":false,"slug":"openebs-plus-red-hat-openshift-and-stackpointcloud-and-ibm-cloud-private-and…."},{"id":104,"title":"How to install OpenEBS on OpenShift?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"22-01-2018","tags":["Kubernetes","Mongodb","Openshift","Percona","Solutions","OpenEBS"],"excerpt":"OpenShift Origin is the upstream community project used in all deployment models provided by Red Hat OpenShift such as OpenShift Online, OpenShift Dedicated, and OpenShift Container Platform.","content":"\n## What is OpenShift Origin?\n\n**OpenShift Origin** is the upstream community project used in all deployment models provided by Red Hat OpenShift such as **OpenShift Online**, **OpenShift Dedicated**, and **OpenShift Container Platform**. **Red Hat OpenShift** is an enterprise container application platform that is based on a core of **Docker** container packaging, **Kubernetes** container cluster management and the **OpenShift Origin** project itself.\n\nFirst, having more than one option sounds confusing, but they clearly differentiate from each other. Here is the summary of all available deployment models to start with OpenShift.\n\n- [**OpenShift Origin**](https://www.openshift.org/) is a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. Origin is open source and all source code for the Origin project is available under the Apache License v2.0 on GitHub. Website and documentation for the Origin project are under [www.openshift.org](https://www.openshift.org/). It is completely free, you can deploy Origin on baremetal, in a VM or on a cloud. This is the option I will focus on this article.\n- [**OpenShift Online**](https://manage.openshift.com/) is Red Hat’s public cloud application development and hosting service. Starter plan is free to use and includes 1 project, 1GiB memory, 1GiB terminating memory and 1GiB storage. Pro Plan costs $50/month and includes 10 projects, 2GiB memory, 2GiB terminating memory and 2GiB storage. Details are available [here](https://www.openshift.com/pricing/index.html).\n- [**OpenShift Dedicated**](https://www.openshift.com/dedicated/index.html) is Red Hat’s managed private cluster offering, built around a core of application containers powered by Docker, with orchestration and management provided by Kubernetes, on a foundation of Red Hat Enterprise Linux. It’s available on the Amazon Web Services (AWS) and Google Cloud Platform (GCP) marketplaces. A complete OpenShift 3 cluster, configured for high availability (HA) with a minimum of 5 masters, infrastructure nodes and 4 application nodes managed by Red Hat costs $48k. Details are available [here](https://www.openshift.com/dedicated/index.html#pricing).\n- [**OpenShift Container Platform**](https://www.openshift.com/container-platform/index.html) (formerly OpenShift Enterprise) is Red Hat’s on-premise private PaaS product.\n\nIn this blog post, I will focus on configuring **OpenEBS** as a **persistent storage** option on the open-source self-managed **OpenShift Origin** and deploy a stateful workload both from CLI and custom catalog template using OpenEBS storage classes.\n\n## Prerequisites\n\n### Hardware\n\n- Minimum two nodes. Recommended four or more (Baremetal, VMs or cloud instances)\n\n### Software components used\n\n- [CentOS 7.x](https://www.centos.org/download/)\n- [OpenShift Origin 3.7+](https://github.com/openshift/origin)\n- [OpenShift-Ansible](https://github.com/openshift/openshift-ansible) (master branch used for installation)\n- [OpenEBS 0.5.1](https://openebs.io/)\n\n**Note:** Make sure the following package dependencies are installed: python, wget, git, net-tools, bind-utils, iptables-services, bridge-utils, bash-completion, kexec-tools, sos, psacct, docker-1.12.6, ansible, pyOpenSS, httpd-tool\n\n### Install OpenShift Origin\n\nFollow instructions from [OpenShift Origin Latest Documentation](https://docs.openshift.org/latest/welcome/index.html) to deploy a multi-node Origin cluster. If you are deploying it for the first time it may be a bit complicated. I plan to post my notes, steps on getting minimum requirements satisfied and have a successful deployment after this post.\n\n### Verify OpenShift Origin deployment\n\nExecute the following commands to verify successful installation.\n\n```\n# oc get nodes\n```\n\nNumber of nodes you see maybe different in your case, but status should looks similar to below showing nodes ready.\n\n```\n# oc get nodes\n NAME STATUS AGE VERSION\n oonode1 Ready 2d v1.7.6+a08f5eeb62\n oonode2 Ready 2d v1.7.6+a08f5eeb62\n oonode3 Ready 2d v1.7.6+a08f5eeb62\n oonode4 Ready 2d v1.7.6+a08f5eeb62\n```\n\n### Configure access permissions\n\nCreate a new admin user with cluster-admin role/permissions and assing password using the following commands:\n\n```\n# oc adm policy add-cluster-role-to-user cluster-admin admin — as=system:admin\n# htpasswd /etc/origin/master/htpasswd admin\n```\n\nLogin as the `admin` user and you will be using default project.\n\n```\n# oc login -u admin\n```\n\nOutput:\n\n```\n# oc login -u admin\n Authentication required for https://oonode1:8443 (openshift)\n Username: admin\n Password:\n Login successful.\nYou have access to the following projects and can switch between them with ‘oc project ’:\n* default\n kube-public\n kube-service-catalog\n kube-system\n logging\n management-infra\n openshift\n openshift-ansible-service-broker\n openshift-infra\n openshift-node\n openshift-template-service-broker\n openshift-web-console\nUsing project “default”.\n```\n\nProvide access to the host volumes which is needed by the OpenEBS volume replicas by updating the default security context (scc). If you miss this step your replicas will fail to deploy.\n\n```\n# oc edit scc restricted\n```\n\nNow set `allowHostDirVolumePlugin: true` and save changes. The file should look like below:\n\n```\n# Please edit the object below. Lines beginning with a ‘#’ will be ignored,\n# and an empty file will abort the edit. If an error occurs while saving this file will be\n# reopened with the relevant failures.\n#\nallowHostDirVolumePlugin: true\nallowHostIPC: false\nallowHostNetwork: false\nallowHostPID: false\nallowHostPorts: false\nallowPrivilegedContainer: false\nallowedCapabilities: []\nallowedFlexVolumes: []\napiVersion: v1\ndefaultAddCapabilities: []\nfsGroup:\ntype: MustRunAs\ngroups:\n— system:authenticated\nkind: SecurityContextConstraints\nmetadata:\nannotations:\nkubernetes.io/description: restricted denies access to all host features and requires\npods to be run with a UID, and SELinux context that are allocated to the namespace. This\nis the most restrictive SCC and it is used by default for authenticated users.\ncreationTimestamp: 2018–01–20T19:39:18Z\nname: restricted\nresourceVersion: “68274”\nselfLink: /api/v1/securitycontextconstraints/restricted\nuid: 9abddec5-fe19–11e7–8d06–005056873c08\npriority: null\nreadOnlyRootFilesystem: false\nrequiredDropCapabilities:\n— KILL\n— MKNOD\n— SETUID\n— SETGID\nrunAsUser:\ntype: MustRunAsRange\nseLinuxContext:\ntype: MustRunAs\nsupplementalGroups:\ntype: RunAsAny\nusers: []\nvolumes:\n— configMap\n— downwardAPI\n— emptyDir\n— hostPath\n— persistentVolumeClaim\n— projected\n— secret\n```\n\nSave changes.\n\n### Install OpenEBS on Origin\n\nThere are few easy ways to install OpenEBS. You can either apply the operator and storage classes direct from the URL or clone the repo and execute from the local copy. I prefer to clone a local copy, but i’ll also give you the other option if you prefer.\n\nClone the latest OpenEBS files and sample application specs using the below command on your OpenShift master node:\n\n```\n# git clone https://github.com/openebs/openebs.git\n# cd openebs/k8s\n```\n\nApply the file two yaml files below:\n\n```\n# oc apply -f openebs-operator.yaml\n# oc apply -f openebs-storageclasses.yaml\n```\n\nAlternative way — If you choose not to copy from the repo you can apply the yaml file direct from the URL below:\n\n```\noc apply -f https://openebs.github.io/charts/openebs-operator.yaml\n```\n\n### Verify OpenEBS deployment\n\nVerify that the OpenEBS provisioner and API server are created successfully and running.\n\n```\n# oc get deployments\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nmaya-apiserver 1 1 1 1 2d\nopenebs-provisioner 1 1 1 1 2d\n```\n\nCheck pods to confirm maya-apiserver and openebs-provisioner.\n\n```\n# oc get pods\nNAME READY STATUS RESTARTS AGE\ndocker-registry-1-b5r7t 1/1 Running 0 2d\nmaya-apiserver-3053842955-xbx8w 1/1 Running 0 2d\nopenebs-provisioner-2499455298–46brm 1/1 Running 0 2d\nregistry-console-1-mrpc9 1/1 Running 0 2d\nrouter-1-bf775 1/1 Running 3 2d\n```\n\nCheck services to confirm maya-apiserver exists.\n\n```\n# oc get service\nNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndocker-registry 172.30.113.229 5000/TCP 2d\nkubernetes 172.30.0.1 443/TCP,53/UDP,53/TCP 2d\nmaya-apiserver-service 172.30.17.113 5656/TCP 2d\nregistry-console 172.30.148.98 9000/TCP 2d\nrouter 172.30.229.239 80/TCP,443/TCP,1936/TCP 2d\n```\n\nCheck service accounts for openebs-maya-operator:\n\n```\n# oc get sa\nNAME SECRETS AGE\nbuilder 2 2d\ndefault 3 2d\ndeployer 2 2d\nopenebs-maya-operator 2 2d\nregistry 3 2d\nrouter 2 2d\n# oc get clusterrole openebs-maya-operator\n\\NAME\nopenebs-maya-operator\n# oc get clusterrolebindings openebs-maya-operator\nNAME ROLE USERS GROUPS SERVICE ACCOUNTS SUBJECTS\nopenebs-maya-operator /openebs-maya-operator default/openebs-maya-operator, default/default\n```\n\nAnd finally verify OpenEBS default storage classes.\n\n```\n# oc get sc\nNAME TYPE\nopenebs-cassandra openebs.io/provisioner-iscsi\nopenebs-es-data-sc openebs.io/provisioner-iscsi\nopenebs-jupyter openebs.io/provisioner-iscsi\nopenebs-kafka openebs.io/provisioner-iscsi\nopenebs-mongodb openebs.io/provisioner-iscsi\nopenebs-percona openebs.io/provisioner-iscsi\nopenebs-redis openebs.io/provisioner-iscsi\nopenebs-standalone openebs.io/provisioner-iscsi\nopenebs-standard openebs.io/provisioner-iscsi\nopenebs-zk openebs.io/provisioner-iscsi\n```\n\nAfter few easy steps we are now ready to deploy workloads on persistent storage provided by OpenEBS. I’ll cover both CLI and Catalog installation through the OpenShift Web Console.\n\n### Install Percona on OpenEBS using OC CLI\n\nUse OpenEBS as persistent storage for a Percona DB deployment by selecting the `openebs-percona` storage class in the persistent volume claim. I will use the sample file available in the openebs repo which I cloned locally in the previous steps.\n\nView the Percona deployment yaml:\n\n```\n# cd openebs/k8s/demo/percona\n# cat demo-percona-mysql-pvc.yaml\n— -\napiVersion: v1\nkind: Pod\nmetadata:\nname: percona\nlabels:\nname: percona\nspec:\ncontainers:\n— resources:\nlimits:\ncpu: 0.5\nname: percona\nimage: percona\nargs:\n— “ — ignore-db-dir”\n— “lost+found”\nenv:\n— name: MYSQL_ROOT_PASSWORD\nvalue: k8sDem0\nports:\n— containerPort: 3306\nname: percona\nvolumeMounts:\n— mountPath: /var/lib/mysql\nname: demo-vol1\nvolumes:\n— name: demo-vol1\npersistentVolumeClaim:\nclaimName: demo-vol1-claim\n— -\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\nname: demo-vol1-claim\nspec:\nstorageClassName: openebs-percona\naccessModes:\n— ReadWriteOnce\nresources:\nrequests:\nstorage: 5G\n```\n\nAs you can see in the YAML file above, `storageClassName` is set to `openebs-percona` which has 2 replicas.\n\nNow, apply the file:\n\n```\n# oc apply -f demo-percona-mysql-pvc.yaml\n```\n\nFinally, verify that Percona is deployed and OpenEBS controller and replica are running:\n\n```\n# oc get pods\nNAME READY STATUS RESTARTS AGE\ndocker-registry-1-b5r7t 1/1 Running 0 2d\nmaya-apiserver-3053842955-xbx8w 1/1 Running 0 2d\nopenebs-provisioner-2499455298–46brm 1/1 Running 0 2d\npercona-1378140207–5q2gb 1/1 Running 0 15mh\npvc-c7a24dc8-ffc7–11e7-a7cd-005056873c08-ctrl-1719480235-xf4t5 2/2 Running 0 15m\npvc-c7a24dc8-ffc7–11e7-a7cd-005056873c08-rep-1550141838-ldm59 1/1 Running 0 15m\n```\n\n### Install MongoDB on OpenEBS using the OpenShift Web Console\n\nLogin to the OpenShift Web Console using the admin credentials we have created earlier.\n\n![OpenShift Origin](https://cdn-images-1.medium.com/max/800/0*-IbP4t-ZgYZx4qh6.png)\n\nClick on **Add to Project** button and select **Import YAML / JSON**.\n\n![Add to project](https://cdn-images-1.medium.com/max/800/0*FEwbuF146LMi7Zsx.png)\n\nCopy the content of [https://raw.githubusercontent.com/openebs/openebs/master/k8s/openshift/examples/v3.7/db-templates/openebs-mongodb-persistent-template.json](https://raw.githubusercontent.com/openebs/openebs/master/k8s/openshift/examples/v3.7/db-templates/openebs-mongodb-persistent-template.json) file and paste into **Import YAML / JSON** window.\n\n![Import Yaml/JSON](https://cdn-images-1.medium.com/max/800/0*d6b0iSD6JG83ad-N.png)\n\nClick on **Create** button, select **Save template** and click **Continue**.\n\n![Add template](https://cdn-images-1.medium.com/max/800/0*14UvCpI6Gf-Q5Pd2.png)\n\nOn the **Template Configuration** window make sure Storage Class is `openebs-standard` and click on **Create**.\n\n![Template configuration](https://cdn-images-1.medium.com/max/800/0*l_agQ7YUPJnqvKkq.png)\n\n![Add template configuration](https://cdn-images-1.medium.com/max/800/0*K8665fQzu2nIGNZh.png)\n\n![Result](https://cdn-images-1.medium.com/max/800/0*E6Vp2d7hqBWtJpzm.png)\n\nYou have successfully deployed MongoDB on a persistent storage provided by OpenEBS.\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/how-to-install-openebs-on-openshift/)_._\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-openshift?"},{"id":105,"title":"Using OpenEBS as a Kubernetes persistent volume","author":"Jimmy Song","author_info":"Developer Advocate at Ant Financial, CNCF Ambassador, co-founder of ServiceMesher community, blog https://jimmysong.io","date":"10-01-2018","tags":["Kubernetes","OpenEBS","Docker","Cloud Native"],"excerpt":"OpenEBS is a containerized block storage written in Go for cloud native and other environments which make the data workloads more reliable in Kubernetes.","content":"\n[OpenEBS](https://www.openebs.io/) is a containerized block storage written in Go for cloud native and other environments which make the data workloads more reliable in Kubernetes.\n\nOpenEBS is open sourced by [MayaData](http://www.mayadata.io/) who is a professional containerized storage company formerly known as CloudByte. Their vision is to make data workloads easy to use in Kubernetes across clouds or on premise.\n\nWe know that [EBS](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://amazonaws-china.com/cn/ebs/&usg=ALkJrhhv8rYmHkvvZS_bPmr_Ca1Wj24SnA) (Elastic Block Storage) is available in AWS, persistent block storage for Amazon EC2 to meet the functional and performance requirements of the most demanding applications, and OpenEBS is its open source implementation.\n\n## Introduction\n\nWith OpenEBS, you can treat containers that have persistent data as you would any other common container. OpenEBS itself is also deployed through containers that support Kubernetes, Swarm, Mesos, Rancher orchestration scheduling, and storage services can be assigned to each pod, application, cluster, or container level, including:\n\n- Data persistence across nodes\n- Synchronize data across available zones and cloud vendors\n- Use commercial hardware and container engines to provide highly scalable block storage\n- Integration with the container orchestration engine, the developer’s application can automatically configure OpenEBS\n- Based on CloudByte’s container-based experience in BSD, we provide users with OpenEBS QoS assurance\n\n## Architecture\n\nThe OpenEBS storage controller itself runs in a container. OpenEBS Volume consists of one or more containers that run microservices. This storage controller function is based on a microservices architecture — the data for each volume is provided by its own set of containers, not by a single monolithic storage controller that provides control for multiple volumes at the same time To provide. This is the essential difference between OpenEBS and traditional storage devices.\n\nThe OpenEBS architecture can be divided into Data Plane (Data Plane) and Control Plane (Control Plane) in two parts:\n\n- Data Plane: Provides data storage for applications\n- Control Plane: Managing OpenEBS Volume Containers, which typically uses the functionality of container layout software\n\n## Data plane\n\nThe following figure shows the architecture of OpenEBS deployed on Kubernetes cluster. Among them, the yellow or orange part is the OpenEBS persistent storage volume, created by Kubernetes’ PVs, implemented using iSCSI, and stored on host nodes or in the cloud (such as EBS, GPD, etc.) depending on where your cluster is deployed. The OpenEBS volume is completely independent of the user’s application life cycle to manage, which is Kuberentes PV in the basic idea.\n\n![OpenEBS Cluster - Data Pane](/images/blog/openebs-data-plane.png)\n\nOpenEBS volumes provide persistent storage for containers with resiliency to system failures and faster access to storage, snapshots and backups. In addition, it provides mechanisms for monitoring usage and enforcing QoS policies.\n\nThe disk that stores the data is called the storage backend and can be a host directory, an attached block device, or a remote disk. Each OpenEBS volume contains an iSCSI target container (represented as openebs-vol1 in the previous figure) and one or more replica containers (openebs-vol1-R1 and openebs-vol1-R2).\n\nThe application pod accesses the storage through the iSCSI target container, which copies the data to all of its replicas. In the event of a node failure, the iSCSI target container starts from one of the remaining online nodes and provides data by connecting to the available replica containers.\n\n**Source**\n\nThe implementation of this section consists of two containers:\n\n- [openebs/jiva](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/jiva&usg=ALkJrhhhCfHb4LkQReHbpayqLJwjwdctgw) : storage control functions, including copy logic\n\n- [openebs/gotgt](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/gotgt&usg=ALkJrhgoXb10SL2TVf8_urB_TIfEVSDBxg) : iSCSI target features used by openebs/jiva\n\n## Control plane\n\nThe OpenEBS control plane is also known as maya. The purpose is to create a hyper-converged OpenEBS that is mounted on a container scheduling engine such as Kubernetes, Swarm, Nomad, etc. to extend the storage capabilities provided by a particular container orchestration system.\n\n![OpenEBS Cluster - Control Plane](/images/blog/openebs-control-plane.png)\n\nOpenEBS’s control plane is also based on microservices, and its services can be divided into the following sections:\n\nContainer layout plug-in, used to enhance the function of the strong container layout framework:\n\n- **Kubernetes Dynamic Configuration** : [openebs-provisioner](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/external-storage/tree/master/openebs&usg=ALkJrhjuOf_IBvwR0NC-g734l_p4Ia14hg)\n- **Kubernetes-dashboard** : [openebs-dashboard](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/dashboard&usg=ALkJrhigRmJSDzmVT_NRMupygPwAM5EX9g)\n- **Extended schema** : Kubernetes-based CRDs (custom resource defination) that store OpenEBS-related configuration data\n\nCluster services provide OpenEBS-specific storage intelligence such as:\n\n- **maya-apiserver** : Contains APIs for performing volume operations that translate requests into container-specific system-specific operations\n- **maya-mulebot** : Use the information collected to suggest optimized layout and event handling tips\n- **maya-connect** : Allows monitoring data to be uploaded to `maya-cloud` for further storage access mode analysis\n\nNode Services, which provide OpenEBS-specific storage intelligence that runs with kubelet, such as:\n\n- **maya-agent** : Includes storage management features\n\nBy using prometheus, heapster, grafana and jaegar for these services, you can add monitoring and tracking capabilities.\n\n**Source**\n\n- [openebs / maya](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/maya&usg=ALkJrhgksSLVDOSt9WRSnCdGdaf4nezkyQ) : All of the specific binary code (non-plugins) is stored in this repository, such as `maya-apiserver` , `maya-agent` , `maya-mulebot` , `maya-connect` , `mayactl` and more.\n- [openebs-dashboard](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/dashboard&usg=ALkJrhigRmJSDzmVT_NRMupygPwAM5EX9g) : A branch of the kubernetes-dashboard project that extends storage capabilities.\n- [openebs-provisioner](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/external-storage/tree/master/openebs&usg=ALkJrhjuOf_IBvwR0NC-g734l_p4Ia14hg) : The OpenEBS K8s Provisioner from the Kubernetes incubator project.\n\n## Install OpenEBS on Kubernetes\n\nBelow we will use the way to install OpenEBS operator, you need to make sure you have already installed iSCSI on your node before installation.\n\n## Prerequisites\n\nOpenEBS relies on iSCSI for storage management, so you need to make sure that you have OpenEBS installed on your cluster.\n\n**Note** : If you are using kubeadm, container-mounted kublet, it comes with iSCSI and does not need to be manually installed. For a kubelet installed directly on the bare metal in binary form, you need to install iSCSI yourself.\n\nThe iSCSI (Internet Small Computer System Interface) is a TCP / IP-based protocol used to establish and manage interconnections between IP storage devices, hosts and clients, and to create storage area networks (SANs ). The SAN makes it possible for the SCSI protocol to be used in high-speed data transmission networks, with block-level data transfer between multiple data storage networks. The SCSI architecture is based on C/S mode and is typically used in environments where devices are close to each other and these devices are connected by a SCSI bus.\n\nOpenEBS needs to use iSCSI as a storage protocol, and CentOS default does not have iSCSI installed, so we need to manually install.\n\nThere are two types of roles in iSCSI:\n\n- **target** : used to provide storage (server)\n- **initiator** : use the stored client (client)\n\nThe following figure in Kubernetes uses iSCSI architecture (Source: [http://rootfs.github.io/iSCSI-Kubernetes/)](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=http://rootfs.github.io/iSCSI-Kubernetes/%25EF%25BC%2589&usg=ALkJrhgk4iuBd1pHB1zGq6XKLwffkSGZew)\n\n![iSCSI-Kubernetes](/images/blog/iscsi-kubernetes.png)\n\nInstalling the iSCSI service is very simple, you do not need additional configuration, just start the service after installation.\n\nExecute the following command on each node node:\n\n```\nyum -y install iscsi-initiator-utils systemctl enable iscsid systemctl start iscsid\n```\n\n## Quick start\n\nRun the OpenEBS service using Operator:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml kubectl apply -f openebs-operator.yaml\n```\n\nUse the default or custom storageclass:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml kubectl apply -f openebs-storageclasses.yaml\n```\n\nMirror used are:\n\n- openebs/m-apiserver: 0.5.1-RC1\n- openebs/openebs-k8s-provisioner: 0.5.1-RC2\n- openebs/jiva: 0.5.1-RC1\n- openebs/m-exporter: 0.5.0\n\n## Test\n\nLet’s use the Example from the official OpenEBS documentation to install the Jenkins test:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/jenkins/jenkins.yml kubectl apply -f jenkins.yml\n```\n\nCheck PV and PVC\n\n```\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0 5G RWO Delete Bound default/jenkins-claim openebs-standard 1h\n$ kubectl get pvc kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\njenkins-claim Bound pvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0 5G RWO openebs-standard 1h\n```\n\nView Jenkins pod:\n\n```\nEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 29m (x2 over 29m) default-scheduler PersistentVolumeClaim is not bound: \"jenkins-claim\" (repeated 3 times) Normal Scheduled 29m default-scheduler Successfully assigned jenkins-668dfbd847-vhg4c to 172.20.0.115 Normal SuccessfulMountVolume 29m kubelet, 172.20.0.115 MountVolume.SetUp succeeded for volume \"default-token-3l9f0\" Warning FailedMount 27m kubelet, 172.20.0.115 Unable to mount volumes for pod \"jenkins-668dfbd847-vhg4c_default(8e2ad467-f1e5-11e7-aa47-f4e9d49f8ed0)\": timeout expired waiting for volumes to attach/mount for pod \"default\"/\"jenkins-668dfbd847-vhg4c\". list of unattached/unmounted volumes=[jenkins-home] Warning FailedSync 27m kubelet, 172.20.0.115 Error syncing pod Normal SuccessfulMountVolume 26m kubelet, 172.20.0.115 MountVolume.SetUp succeeded for volume \"pvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0\" Normal Pulling 26m kubelet, 172.20.0.115 pulling image \"sz-pg-oam-docker-hub-001.tendcloud.com/library/jenkins:lts\" Normal Pulled 26m kubelet, 172.20.0.115 Successfully pulled image \"sz-pg-oam-docker-hub-001.tendcloud.com/library/jenkins:lts\" Normal Created 26m kubelet, 172.20.0.115 Created container Normal Started 26m kubelet, 172.20.0.115 Started container\n```\n\nStart up successful. The Jenkins configuration uses **NodePort** mode access and now accesses the NodePort of Jenkins service for any node in the cluster.\n\n## Reference\n\n- [OpenEBS Documentation](http://openebs.readthedocs.io/)\n- [CentOS 7.x 下配置 iSCSI 网络存储](http://blog.csdn.net/wh211212/article/details/52981305)\n- [Configure iSCSI Initiator](https://www.server-world.info/en/note?os=CentOS_7&p=iscsi&f=2)\n- [https://www.openebs.io/](https://www.openebs.io/)\n- [https://github.com/openebs/openebs](https://github.com/openebs/openebs)\n- [Data Scientists adopting tools and solutions that allow them to focus more on Data Science and less on the infrastructure around them](/blog/data-scientists-adopting-tools-and-solutions-that-allow-them-to-focus-more-on-data-science-and-less)\n- [RHEL7: Configure a system as either an iSCSI target or initiator that persistently mounts an iSCSI target.](https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistently/)\n\nOriginal page: [https://jimmysong.io/posts/using-openebs-as-kubernetes-persistent-volume/](https://jimmysong.io/posts/using-openebs-as-kubernetes-persistent-volume/)\n\nTranslated from Chinese to English by Google Translate\n","notHasFeatureImage":true,"slug":"using-openebs-as-a-kubernetes-persistent-volume"},{"id":106,"title":"In 2018 - IT dreams deferred finally achieved?","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"09-01-2018","tags":["Containerization","Docker","Kubernetes","OpenEBS","Storage"],"excerpt":"At MayaData, we believe we, and others are building the foundation for a much longer cycle of software-centric innovation thanks to proactively eliminating sources of lock-in.","content":"\n# **_…..Dreams deferred_**\n\nMany of us in the infrastructure business have been forced by experience to lower our expectations of what is possible. While we’ve all dreamed for decades of a world in which software just works — and delivers value where and how it is needed — we’ve been disappointed again and again.\n\nWe have seen open systems that, over time, became increasingly proprietary with Unix diverging into proprietary camps.\n\nWe’ve seen SQL go from a fascinating research project to a broadly deployed standard to, with the help of stored procedures and truly nefarious licensing, a source of lock-in dominated by one company and one eccentric multi-billionaire.\n\nWe’ve seen a vision of Java as a cross infrastructure abstraction layer bloom and wither.\n\nAnd of course, we’ve seen virtual machines offer the promise of cross infrastructure mobility only to fall prey to the rest of the stack and proprietary business models and incorrect levels of abstraction.\n\nOver time the result has been infrastructure by silos, with each silo — security, storage, networking and compute — dominated by proprietary solution providers that over time sought to provide the entire stack to drive up their sales, even if doing so meant increasing the friction for users seeking to combine best of breed solutions.\n\n# **_On the other hand…._**\n\nAll along technological progress has continued. Allowing for new possibilities.\n\nWe’ve even seen — finally — broadband make its way into the United States so that more and more we can cost effectively access the cloud (yes, the loss of net neutrality seems to put this at risk for at least consumers and new entrants).\n\nAnd intra data center networking has gotten insanely fast — which is crucial if we are to run workloads in a flexible manner.\n\nAnd arguably the best example of innovation at scale in modern business — Amazon — focusing on the right persona — the developer — and raising the bar massively for all of us in infrastructure.\n\nAnd perhaps most importantly, the Open Source community, which as Richard Stallman and others have pointed out predates the commercial software world and which some have called the world’s first social network, grew to become an undeniable force.\n\nAnd pulling all the positive forces together — DevOps and microservices. DevOps as a cultural movement and approach to building and running software at scale PLUS an emerging understanding of how to run systems via microservices as explained by the [12 factor approach](https://www.12factor.net/) and elsewhere led to countless examples of “software eating the world.”\n\n# **But …. What about lock-in?**\n\nSo as the above suggests, one theme in the story of innovation in IT over the years has been breakthrough technologies, and business models, enabling fundamentally better software delivered more easily to users. And as one approach came to predominate, proprietary approaches over time led to more “rent seeking”, where leading vendors extracted more value from their users and slowed their innovation. And this stagnation leads to pent up demand for better approaches — triggering the next cycle.\n\nWell — what about this time?\n\nKubernetes has emerged in part because it promises a world more free from lock-in to AWS and other clouds. Could it be that we have collectively learned enough from all the boom and bust cycles to know what is good for us?\n\nCould be — the signs are incredibly promising as all the cloud vendors and RedHat and Cloud Foundry and Docker and Mesos have all embraced Kubernetes as the standard control plane. This means that you are no longer locked-in by the control plane logic and should be able to move your applications from cloud to cloud and from on premise to off. Crucially — Kubernetes itself is open source and all the major vendors have pledged to not fork it; so it shouldn’t be _too_ bad to move from one vendor supporting Kubernetes to another.\n\n_…. but what about data?_ Without data mobility all you can move is the stateless components of your applications — provided you address having those components able to access your store of state.\n\n# **And your data remains largely locked-in**\n\n_Locked into_ proprietary vendors.\n\n_Locked into_ underlying systems that are sources of risk and that themselves are resolutely monolithic.\n\nI harken back to a speech Randy Bias gave at one of the OpenStorage summits I helped host back in 2010 about `blast radius`. The basic idea is that microservices dramatically reduce the blast radius of any single outage; conversely putting all your state in a shared storage system is, by comparison, an anti-pattern. When your shared storage dies or slows down unexpectedly perhaps due to a rebalancing, so does your entire environment. So much for being built for failure!\n\nS3 for non performant data and EBS for performant data have become defacto standards. They are easy, they “just work”, and — crucially — they put the responsibility for the configuration, care and feeding of state in the hands of the teams that also control the microservices.\n\nThe only problem is that it is _hard_ to move your data from these AWS services to other solutions without a lot of work that frankly software development teams don’t have the time or inclination to invest. I see the lock-in that results as the TBs pile up treated much as technical debt is treated — it is annoying and yet it is much less important than getting valuable capabilities in the hands of end users.\n\nAnd putting all your data in a scale-out software solution running on these clouds only makes the issue worse. Now you have the blast radius issue and you have your data stored in a solution that cannot be stretched across clouds. Two sources of lock-in and at least twice the effort!\n\nIt might be worth remembering that networking, security and compute are all becoming both infrastructure services delivered as services **to** today’s microservice environments and **are themselves also microservice based services**. Take a look at Project Calico for instance. Or at Kubernetes itself.\n\n**Nobody says — hey, Kubernetes is just a black box that sits to the side and so it needn’t be a bunch of microservices. But not storage. Storage somehow gets a pass. It gets to live with aged architectures and typically aged business models.**\n\n## Which raises the question: What if storage was itself delivered as microservices and orchestrated by Kubernetes?\n\nFor the purpose of this exercise, **assume** it were possible to make storage a set of capabilities delivered as microservices with the controller running on containers.\n\nYou’d probably agree that such an approach would have some benefits including:\n\n**Familiarity:**\n\n- If storage is delivered as microservices within Kubernetes then if you know how to run Kubernetes then you know how to run the storage.\n- Perhaps more importantly, you are familiar with the failure domain. You lose a storage controller — well, you just lost a stateless container that itself simply provides services and pointers towards the underlying data. Your data is always safe in multiple locations and your storage system itself is resilient (at least the way OpenEBS is architected with the use of atomic transactions).\n\n**Granularity:**\n\n- As mentioned above, the defacto standard approach to delivering storage is to use AWS itself with each team organized around one or more microservices having their own approach to EBS for performant storage and S3 for blobs of data.\n- Using a shared storage system runs counter to this approach and cuts these teams out of the loop. They are back to lobbying central IT as one of hundreds or even thousands of workloads with particular desires as to how storage should be configured. And, yes, those configurations matter. And, actually, they are impossible to get right. We’ve talked about that in the past including at Meet-ups: [https://www.slideshare.net/MattBaldwin3/containerized-storage-for-containers-why-what-and-how-openebs-works](https://www.slideshare.net/MattBaldwin3/containerized-storage-for-containers-why-what-and-how-openebs-works)\n\n![What move the data and configs next to the app](/images/blog/what-move-the-data-and-configs-next-to-the-app.png)\n\n**Performant:**\n\n- This being a storage blog, it is worth reiterating the point that shared storage is inherently less performant these days than direct attached or DAS. That is a fairly new reality. It used to be that DAS was really slow disk and the way to get IOPS was to stripe across a bunch of faster disks. That was a primary driver for shared storage. Imagine that — at one time CEPH would have been faster than the underlying hardware! How times have changed.\n- Our CTO, Jeffry Molanus does a good job walking through how the landscape of performance has changed why this and other changes now favor what we call “container attached storage”:\n- [https://openebs.io/blog/not-yet-another-distributed-storage-system](/blog/not-yet-another-distributed-storage-system)\n\n**Natively cross cloud — with the help of metadata and routing services:**\n\n- What is perhaps least well appreciated about the potential of treating storage as a service delivered via microservices is that, correctly engineered, this means that data itself can be served as a service in the background across underlying clouds.\n- The first prerequisite is that the controller itself runs in a container or set of containers.\n- The second prerequisite is that the controller performs its magic in the user space so that the container does not need to be a special build and so that the system can perform.\n- Third, there needs to be the management of metadata to see where the data is versus the workloads. Kubernetes can help here as it expands.\n# **TL;DR:**\n\nSo, in short, this time perhaps it really _is_ different.\n\nThis time we “won’t get fooled again” (gratuitous old guy music reference :)).\n\nThis time we _will_ address the sources of lock-in not just at the controller plane via Kubernetes but also at the data layer. And in so doing we will avoid ending the the cycle of innovation prematurely. Perhaps it goes without saying — only an open source solution like OpenEBS that is widely accepted and easy to fork if needed can help free us from the risk of cloud lock-in without adding yet another source of lock-in.\n\nAnd we can address lock-in while respecting and extending the patterns we know are working including: every team controlling their infrastructure themselves, the elimination of single points of failure (aka “storage blast radius”), and allowing Kubernetes to control more and more of the environment, leaving the developers to focus on capabilities that add value to their end users.\n\nIn short, at MayaData we believe we and others are building the foundation for a much longer cycle of software-centric innovation thanks to proactively eliminating sources of lock-in.\n\nPlease help this reality come true by providing us feedback on [OpenEBS](http://www.openebs.io/) and [MayaData](http://www.mayadata.io/) or see us on the Kubernetes storage SIG where we are trying to be helpful as well.\n","notHasFeatureImage":false,"slug":"in-2018---it-dreams-deferred-finally-achieved?"},{"id":107,"title":"Austin KubeCon — Persistent Storage Round-up and Looking beyond!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"08-01-2018","tags":["Kubecon","Kubernetes","Persistence","Storage Containers","Updates"],"excerpt":"Kubernetes Clusters are up and running at the push of a button or even better by talking to your favorite bot.","content":"\nTL;DR\n\nThis rather has become a long post as I drafted it and incorporated feedback from community members. So, in short:\n\n- 2017 saw Kubernetes being crowned as the de-facto container orchestration engine. And from the storage perspective, containerized storage makes its presence felt.\n- 2018 — The reign of Kubernetes continues. Containerized Storage gains momentum with renewed focus on manageability of persistence workloads\n\n—\n\nKubernetes Clusters are up and running at the [push of a button](https://twitter.com/muratkarslioglu/status/941399154714066944) or even better by [talking to your favorite bot](https://www.youtube.com/watch?v=07jq-5VbBVQ).\n\n![Push a button](https://cdn-images-1.medium.com/max/800/1*oz5esyJvsb5zBIaoyDKUeQ.png)\n\nsource: Containerized Storage for Containers — session at Kubernetes Meetup [https://t.co/tdQaOue5w8](https://t.co/tdQaOue5w8)\n\nBut just about a year ago when we started envisioning OpenEBS — Containerized Storage for Containers — to be orchestrated by Kubernetes, setting up a cluster took a good three days. Phenomenal progress by the Kubernetes Community in 2017 — from Kubernetes — the Hard-Way to making it Child’s play!\n\nIf you were at KubeCon, you would have definitely been caught up in or at least glimpsed the Euphoria around Kubernetes. Kubernetes, almost feels like Noah’s Ark right now — you are either in or wait to perish. A little exaggerated, I know, but only a little.\n\n_Every Technology and Cloud Service Provider are now providing or planning to provide a container service using Kubernetes and almost every infrastructure provider is looking at putting themselves on the map of Kubernetes._\n\n![Cloud Native Landscape](https://cdn-images-1.medium.com/max/800/1*YJIF6xBEPL1WpVgOK4VV0Q.png)\n\n[https://raw.githubusercontent.com/cncf/landscape/master/landscape/CloudNativeLandscape_latest.png](https://raw.githubusercontent.com/cncf/landscape/master/landscape/CloudNativeLandscape_latest.png)\n\nAnd why not!\n\nKubernetes has reached the level of maturity that can be used with ease in controlled environments and at the same time, has gained tremendous strength from a community that is not afraid to re-engineer or re-architect the core components. The popularity of Kubernetes is enabling many meta-kubernetes projects like — kubespray, stackpointcloud, kubeless, heptio ksonnet, heptio ark, etc. And with these new projects and possibilities, many _Kubernetes — blue ocean — companies are on the rise!_\n\n—\n\nI am very bullish that Kubernetes is that magical ingredient that will renew the focus on HumanOps!\n\nOne inevitable aspect of being an infrastructure operations admin is to be prepared for smooth operations, scaling, maintaining and recovering from faults and disasters — which usually tend to put a lot of unwarranted pressure on the admins when dealing with their own management (business impact) and the vendors whose technology they used to build a “black-box” infrastructures. A “black-box” infrastructure that doesn’t comply with what they were told and assumed would do — and involves talking to people building those black-boxes often crossing company borders, leading into non-technical calls about blaming who is at fault. Such stressful conditions also exist within an organization where there is a crunch of resources.\n\nKubernetes and the meta-kubernetes projects are helping administrators build — what I call “white-box” infrastructures. Often professed and hardly-prevailed aspect of Infrastructure is the [_HumanOps_](https://blog.serverdensity.com/humanops/) _— \"_ technology affects the well being of humans just as humans affect the reliable operation of technology *\"* — which can be achieved by building “white-box” infrastructures that are easy to operate and reduce the dependency on specialists that tend to be over-worked in an organization. _The “white-box” infrastructures are built with API-driven Open Source Micro-Services._\n\nThe key to the widespread adoption of Kubernetes in such a short time is the inclusive nature of it, which was well captured by this slide from the KubeCon:\n\n![Extensibility](https://cdn-images-1.medium.com/max/800/1*IXods_RnXRco2z7UcngePw.png)\n\n[https://schd.ws/hosted_files/kccncna17/ac/KubeCon_2017\\_-_Kernels_and_Distros.pdf](https://schd.ws/hosted_files/kccncna17/ac/KubeCon_2017_-_Kernels_and_Distros.pdf)\n\n### **_Kubernetes is more than an orchestration engine → It is the new kernel for building clustered infrastructures._**\n\nI consider this shift towards making Kubernetes a Kernel that can be extended by custom solutions that can be downloaded and installed as a true enabler for driving innovation — which is inline with the Psyche of keeping “Community First and Company Next!”.\n\n—\n\nThis past year, saw the rise in user awareness for securing Containers. Different teams started tackling this issue from different perspectives — from providing secure container runtime like Kata Containers to using different types of Service Meshes to better access control and more.\n\nService Mesh was definitely a buzzword in 2017! The options — linkerd, envoy, istio and conduit — all of which are (or will be) accepted into CNCF sooner or later, provide a glimpse of interesting trend. For instance, [conduit](https://buoyant.io/2017/12/05/introducing-conduit/) is from the same team that built Linkerd. Conduit provides similar capabilities like Linkerd for managing the communication between micro-services, but seems better suited for Kubernetes environment that can run across clouds and with low resource constraints.\n\nLike Service Mesh, the other infrastructure components — logging, monitoring, tracing, and networking are all being containerized (re-engineered) to work well with Kubernetes primitives (resources, pods, policies, federations, labels, taints, tolerations, affinity and anti-affinity, CR, CRDs, Custom Controllers, etc.)\n\n### **_Kubernetes has become a powerful set of nuts and bolts, that is changing the way people should think about infrastructures and how systems are built._**\n\n—\n\nStorage is no different. How data is stored and managed is also being transformed by the possibilities afforded by Kubernetes. Like Service Mesh of initial days, a lot of incumbent storage vendors are providing a patched (which some view as cloud washed or container plugged) solutions that will result in operations and developers spending endless hours firefighting to make them work with cloud native environments.\n\nThe key for any infrastructure component to be called container-native will be characterized by being hardware agnostic and usable at scale! The past few months, there is an active workgroup team grappling with defining — [_Cloud Native Storage (WIP White Paper by CNCF Storage Workgroup)_](https://docs.google.com/document/d/1cJLgOAIWbi-Ya27BY7mjH61yoO3oWcO5tOZYteaDVgI/edit#heading=h.ik4inq9mv6b4)\n\nWhile deliberations are ongoing about what Cloud Native Storage is, which I think will finally be about users adoption, the talks at KubeCon suggest the community sees three distinct storage options for Kubernetes:\n\n- Persistence Volumes from External Storage Providers\n- Local/Ephemeral Storage for Containers\n- Containerized Storage for Containers\n\n—\n\n**Persistence Storage from External Storage Providers**\n\nMost of the cloud providers and incumbent storage vendors want the users to opt for this option where storage is connected via in-tree or out-of-tree dynamic volume provisioners. Many vendors are coming together in helping shape the CSI [(Container Storage Interface)](https://github.com/container-storage-interface/spec), and the initial implementation are slated to get into beta stage in early 2018. There are constant improvements — or strides — being made in storage workflow automation via controllers and `kubectl` — dynamically provisioning volumes, resizing, and snapshots.\n\nI spoke to a number of storage users at KubeCon, including the team at GitHub who are at the forefront of putting Kubernetes in production. The users are still very wary with the state of storage w.r.t using the PVs to connected storage and the amount of work involved in rewriting their operational scripts and playbooks.\n\nAnother issue I heard users talk about that puts them off NAS or SAN — and this was a little surprising as I’ve spent years building a unified storage system that in some environments is really fundamental to the architectures of private clouds and hosting environments — is that they think shared underlying storage does not fit a microservices architecture. Of course, if you read the 12 Factor definition it talks about storage if at all as an attached resource. However — it also is clear from 12 Factor approaches that _dev should be the same as possible as production and that the same people should be doing coding and deploys._ That’s just not the world of external arrays with special teams running storage and different arrays for dev, test, staging, and production.\n\nIt is also worth noting and taking time to understand that these options of connecting to network storage have been around for more than couple of years, and the fact that Stateful workloads on Kubernetes aren’t yet as prevalent says something about user acceptance of the approach! _Users are waiting for better options to be made available — like the support for local storage or something else — but not NAS!_\n\nLearning from the HBO team that was streaming GoT using Kubernetes, it is interesting to see a solution like Rook being used on top of EBS, while EBS is provided as PVs themselves.\n\n![Telemetry](https://cdn-images-1.medium.com/max/800/1*Zl5PPYzJpDZoXrK7DCL_0w.png)\n\n[https://www.youtube.com/watch?v=7skInj_vqN0](https://www.youtube.com/watch?v=7skInj_vqN0)\n\nRook also presented a pretty interesting study against using PVs from external storage to Pods in their talk [here](https://schd.ws/hosted_files/kccncna17/b3/Cloud%20storage-2.pdf). This is inline with what the teams at [PortWorx](https://portworx.com/ebs-stuck-attaching-state-docker-containers/), [StorageOS ](https://schd.ws/hosted_files/kccncna17/ca/2017-12-8-persistent-storage.pdf)and OpenEBS have been advocating as well.\n\n- Make static assignments of disks (physical or virtual) to nodes and use them as local storage — avoid detaching/attaching disks from nodes\n- As long as the applications can take care of replication and sustain longer downtime for nodes and cluster rebuilding times — use local PVs with the storage provisioned in the previous steps.\n- For workloads that don’t inherently support replication, snapshots, etc. use a containerized storage option.\n\n_I am a firm believer in CSI and what it was set to achieve and has already accomplished— Open Standard for interfacing with Storage. Something which SNIA should have done and couldn’t do in past two decades of my experience. OpenSDS seems to be an effort in that direction by SNIA, but is being received with the same cold response from vendors and in turn the community. FWIW, REX-Ray is also playing in the same space._\n\nAt the moment, the focus for CSI is on (simplifying a bit) provisioning and de-provisioning volume, but albeit a good start. But is it enough for the users to start using it? There was an interesting observation made in the F2F storage workgroup meeting at KubeCon that CSI discussions are mostly driven by vendors. Where are the users? Can we say that vendors represent the users, because they interact with their users?\n\nComing from a operational background, for me to consider using CSI to connect with external storage systems, CSI requires to evolve and include API for Day 2+ Operational Usecases that involve — ease of debugging, snapshots, backups, migration and most importantly, a unified monitoring system of the Kubernetes Clusters and the Storage Systems.\n\nDon't get me wrong. We need storage, lots and lots of it and it will be served from external storage systems — cloud (EBS, GPD, etc., ) or on-premise SAN/NAS products. But these external storage systems weren’t designed to be used for micro-services environment but rather to provide volumes to Nodes (physical or virtual) that are long running and are not subject to rapid connects, disconnects and migrations.\n\n_I believe in the long run we will be using CSI with these external storage for what they were designed for — mainly to provision storage to the nodes rather than Pods._\n\n—\n\n**Local/Ephemeral Storage for containers (aka Direct Attached Storage — DAS)**\n\nKubernetes keeps improving the capabilities for managing the local/ephemeral storage. The recent advancements include:\n\n- Support for attaching [block devices](https://schd.ws/hosted_files/kccncna17/8e/Mitsuhiro_Tanino_Block_Volume_KC_CNC_NA17.pdf) to pods\n- Support for enforcing policies or [resource limits for ephemeral storage](https://schd.ws/hosted_files/kccncna17/3e/Kubecon_localstorage.pdf)\n- Enhance the UX for using [local storage for PVs](https://schd.ws/hosted_files/kccncna17/3c/2017%20Kubecon%20Storage%20-%20FINAL.pdf) — dynamic provisioning, hook into the scheduler for pods requiring local storage PVs etc.,\n\n_When using local storage for PVs, the applications using these PVs need to also own up some of the features like — data consistency, replication, snapshots, etc., that are typically taken care of by the storage controllers._\n\n_One of the ongoing issue with using the local storage in clouds are the quirks of disconnecting and connecting the disks to different instances. The local storage is really meant for using storage that is tied to the node — either physically inserted or hardwired to a VM instance._\n\n—\n\n**Containerized Storage for Containers (aka Container Attached Storage — CAS)**\n\nThe appeal for fully containerized storage for containers is in the possibilities that it opens up to the DevOps administrators who are interested in building on-demand programmable infrastructures, which include:\n\n- storage can be observed down to the bit using the same set of tools they use to monitor their compute and network.\n- storage can also be secured using the same tools used to secure application pods\n- storage can be made policy driven similar to networks\n- storage can be programmed and versioned — made an integral part of the work flows for developers and operations administrators\n- storage can also use federation features for cloud migration similar to application pods.\n\n_StorageOS presented at KubeCon on what we call Container Attached Storage — and on how to select which storage approach for which workload and environment. It was a good talk — slides are here: _[talk](https://schd.ws/hosted_files/kccncna17/ca/2017-12-8-persistent-storage.pdf)\n\n_Kubernetes can provide an unified infrastructure layer to the applications by pooling together nodes with compute, network, and **storage as well**._\n\nKubeCon showcased a demo of launching [glusterfs in containers](https://schd.ws/hosted_files/kccncna17/7b/KubeRunningYourStorage1208.pdf). While this is feasible, it might put some hard requirements on the amount of RAM and CPU required for running the software optimized for running in the nodes in containers.\n\nTo be container-native storage, the storage software needs to be broken down into micro-services, just like how Kubernetes runs using micro-services. There has to be greater flexibility provided to the developers and operations to run seamlessly on their choice of hardware!\n\nOpenEBS does just that! OpenEBS provides all the enterprise grade storage features by its open-source containers that can run anywhere. _No kernel dependencies and vendor lock-in._ A typical data path using the OpenEBS Containers is as follows:\n\n![Stateful Apps using OpenEBS Volumes](https://cdn-images-1.medium.com/max/800/1*Ifsa-k-q4EnO7Fpg7E6jLA.png)\n\n[https://github.com/openebs/openebs/blob/master/contribute/design/OpenEBS%20Architecture%20and%20Design.pdf](https://github.com/openebs/openebs/blob/master/contribute/design/OpenEBS%20Architecture%20and%20Design.pdf)\n\nOpenEBS can consume any storage connected to the node and provide enterprise grade storage features (like snapshots, replication, data-consistency, cross-cloud migration, etc.) to Stateful workloads.\n\n2017 saw a steep rise in the community for building OpenEBS with users evaluating it for different types of storage workloads from Cassandra, Minio to MySQL and some users also rolling out services to their customers using Kubernetes and OpenEBS. _I am looking forward to more application work-flow focused automation of Stateful workloads using OpenEBS in 2018._\n\n—\n\nManaging Storage in an enterprise environment — whether it is cloud or on-premise has to be as seamless as interacting with your favorite bot! I know it is a bit far fetched, but it is definitely going to happen in 2018 with companies like MayaData leading from the front!\n\n2017 saw some major improvements to storage in Kubernetes, but there is a lot more to look forward to in 2018!\n\n- CSI spec will mature to encompass all the storage API and will be adopted by a large percentage of storage vendors.\n- Improved debuggability/observability of PV — Metrics and Alerts etc.\n- Make PVs accessible via namespaces and RBAC and extend the Policies to involve HumanOps!\n- Further improvements to resource constraints from the IOPS perspective\n- Support for host-supported file system types to be used on top of local storage\n\n_Programmable and Predictable Infrastructures are what the developers need while the administrators are looking for building infrastructures that can be easily versioned, built, and deployed anywhere — where the economics makes sense._\n\n—\n\nI take tremendous pride in having been associated with MayaData team that is at the forefront of making Storage Operations fade away by extending Kubernetes with containerized storage for containers.\n\nYour participation will shape and accelerate the movement of Stateful Workloads on Kubernetes. Do join us on Slack on either [Kubernetes sig-storage](http://slack.k8s.io/) or [OpenEBS users](http://slack.openebs.io/) or join the [CNCF storage events](https://calendar.google.com/calendar/embed?src=linuxfoundation.org_o5avjlvt2cae9bq7a95emc4740%40group.calendar.google.com)!\n\nLooking forward to an exciting 2018 for the Stateful Workloads on Kubernetes!\n","notHasFeatureImage":true,"slug":"austin-kubecon-—-persistent-storage-round-up-and-looking-beyond!"},{"id":108,"title":"Install OpenEBS using StackPointCloud Trusted Charts?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"07-01-2018","tags":["Digital Ocean","Helm","Kubernetes","Solutions","StackPointCloud"],"excerpt":"What is StackPointCloud Trusted Charts?","content":"\n#### What is StackPointCloud Trusted Charts?\n\n[StackPointCloud](https://stackpoint.io/) (SPC) introduced a concept of Trusted Charts, a list of validated [Helm](https://helm.sh/) Charts provided by its partners to quickly spin up a solution in a [Kubernetes](https://kubernetes.io/) cluster. Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application.\n\nPreviously, I wrote about few different ways of getting OpenEBS up and running on different cloud vendors. Using Helm Chart is one of the available options to deploy OpenEBS. OpenEBS Helm Charts were available since v.5.0 both on [Github](https://github.com/openebs/openebs/tree/master/k8s/charts/openebs) and as a [packaged chart](https://openebs.github.io/charts/). Recently SPC included OpenEBS into their Trusted Charts repo and made it one-click easy for its customers.\n\nSPC Trusted Charts currently offers 23 solutions including databases, CI/CD, monitoring, storage and ingress solutions. Here is the list of Trusted Charts:\n\n### CI/CD\n\n- [Jenkins](https://jenkins-ci.org/)\n- [GitLab Runner](https://docs.gitlab.com/runner/)\n- [Spinnaker](https://www.spinnaker.io/)\n\n### Databases\n\n- [CockroachDB](https://www.cockroachlabs.com/)\n- [Crunchy PostgreSQL Operator](https://github.com/CrunchyData/postgres-operator)\n- [Patroni](https://github.com/turbonomic/kubeturbo)\n- [Redis](https://redis.io)\n- [RethinkDB](https://www.rethinkdb.com/)\n- [MongoDB Replica Set](https://docs.mongodb.com/manual/replication/)\n\n### Ingress/Proxy/Load Balancer\n\n- [Nginx Ingress](https://github.com/kubernetes/ingress-nginx)\n- [Traefik](https://traefik.io/)\n\n### Messaging\n\n- [Kafka](https://kafka.apache.org/)\n- [Rabbitmq](https://www.rabbitmq.com)\n\n### Storage\n\n- [OpenEBS](https://openebs.io/)\n- [Minio](https://www.minio.io/)\n- [Etcd-operator](https://github.com/kubernetes/charts/tree/master/stable/etcd-operator)\n\n### Others\n\n- [Grafana](https://grafana.com/)\n- [Keel](https://keel.sh/)\n- [Kube-lego](https://github.com/jetstack/kube-lego)\n- [Kubeturbo](https://github.com/turbonomic/kubeturbo)\n- [Memcached](https://memcached.org/)\n- [Tensorflow Inception](https://github.com/tensorflow/models/tree/master/research/inception)\n\nI’ll go through the quick steps of deploying OpenEBS.\n\n### Prerequisites\n\nMinimum requirements for deploying your Kubernetes clusters on StackPointCloud:\n\n### Cloud Provider Account\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) or\n- [DigitalOcean](https://www.digitalocean.com)\n\n### Deploy a New Kubernetes Cluster\n\nFirst, go to [stackpoint.io](https://stackpoint.io/) and click on **Launch a Cluster** button to start your free trial.\n\n![Universal control plane for managed Kubernetes](https://cdn-images-1.medium.com/max/800/0*0cB3ttYmslFZgH1h.png)\n\nThen choose your cloud provider. In this example, I will use **Digital Ocean**.\n\n![Choose cloud provider](https://cdn-images-1.medium.com/max/800/0*21G24JgfuqlR6snZ.png)\n\n### Configure Access to Digital Ocean\n\nOn the next screen, we need to configure our provider. You need to provide Digital Ocean API Token and optionally your SSH Key.\n\n![Configure your provider](https://cdn-images-1.medium.com/max/800/0*wDcMg-_HTjIOFvgb.png)\n\nClick on **Add API Token** button.\n\n![Add API token](https://cdn-images-1.medium.com/max/800/0*53wGtQ7eUt18u6pS.png)\n\nAfter you add your credentials, click on **Submit**.\n\n### Configure K8s Cluster\n\nOn “Configure your cluster” page click the edit button on **Distribution** and choose **Ubuntu 16.04 LTS**.\n\n![Configure your cluster](https://cdn-images-1.medium.com/max/800/0*NvtnryAA8GNi-fyN.png)\n\nChange the **Cluster Name** something meaningful like **OpenEBS Demo**.\n\n![Enter cluster name](https://cdn-images-1.medium.com/max/800/0*LTa6zBooJdTsyqss.png)\n\nLeave everything else as default and click on **Submit**.\n\nIn about 10–15 minutes you will get your new cluster deployed.\n\n### Adding OpenEBS to Your Kubernetes Cluster\n\nFirst, make sure your cluster and all nodes are up.\n\nOn the **Control Plane** tab click on your recently created cluster.\n\n![Control plane](https://cdn-images-1.medium.com/max/800/0*RHQ9LbyxydjHkJSk.png)\n\nOnce the Kubernetes cluster is up on Digital Ocean with functional Helm, scroll down to the **Solutions** tab and click on **Add Solution** button.\n\n![Add Solution](https://cdn-images-1.medium.com/max/800/0*sH0lzv23vHonV5Zk.png)\n\nClick on **Add Solutions**, and select **Trusted Charts**.\n\n![Select charts](https://cdn-images-1.medium.com/max/800/0*V6iP5PzNAzFk4sME.png)\n\nFrom the list above select **OpenEBS**.\n\n![OpenEBS namespace](https://cdn-images-1.medium.com/max/800/0*CJkPrkJCS9Fp_GXu.png)\n\n**Release Name** is randomly generated every time. If you want to use OpenEBS example workloads provided in OpenEBS repos without any modification then use `default` as **NameSpace**. Otherwise, you need to modify the namespace for workloads you deploy and make sure to use the same name.\n\nClick on **Install** to deploy OpenEBS on your cluster.\n\n**Note:** Default settings assume that RBAC is enabled. If you disabled RBAC while you are configuring your provider previously then set `rbacEnable: false` otherwise use default values.\n\nState field should be green after OpenEBS is successfully added.\n\n![OpenEBS successfully added](https://cdn-images-1.medium.com/max/800/0*HzCZp3Z5LbT3Hsrh.png)\n\nNow your cluster is ready; you can run your workloads on `openebs-standard` and other predefined storage classes.\n\nTo confirm, click on **Kubernetes Dashboard**. This will bring up your Kubernetes Dashboard UI in a new window. You will find all predefined OpenEBS **Storage Classes** here under **Storage Classes** section.\n\n![Kubernetes Storage Classes](https://cdn-images-1.medium.com/max/800/0*mNU-nhwvNy9UB0W5.png)\n\nNow you are ready to deploy your stateful workloads.\n\nTake a look at my previous articles on step-by-step instructions for deploying few popular stateful workloads such as [Cassandra](http://containerized.me/how-to-deploy-a-cassandra-cluster-ring-on-kubernetes-openebs/), [Jenkins](http://containerized.me/how-to-deploy-jenkins-on-kubernetes-openebs/), and [Postgres](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/) on OpenEBS persistent storage.\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/install-openebs-using-stackpointcloud-trusted-charts/)_._\n","notHasFeatureImage":false,"slug":"install-openebs-using-stackpointcloud-trusted-charts?"},{"id":109,"title":"OpenEBS 0.5 enables Storage Policies for Kubernetes Persistent Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"30-11-2017","tags":["Container","Open Source","Storage Containers","Kubernetes","Updates"],"excerpt":"Personally, it is very exciting and enriching to see the growth of the OpenEBS project — from its capabilities, contributors and community perspective!","content":"\n\nPersonally, it is very exciting and enriching to see the growth of the OpenEBS project — from its capabilities, contributors and community perspective!\n\nI believe the metric to measure the success of OpenSource Projects is determined by the number of users and the conversion ratio of users turning into contributors. The past couple of months (thanks to campaigns like Hacktoberfest and the ongoing OpenEBS Hackfest), we have seen a steep rise in the number of contributors and contributions to the OpenEBS project.\n\n![OpenEBS contributors](https://cdn-images-1.medium.com/max/800/1*BMOr9ULh_7KnM6k8aUj9hw.png)\n\nIn almost all the interactions we had with the user community, we are seeing a clear resonance of the value proposition that OpenEBS brings to the [DevOps teams managing systems with large number of micro-services](https://twitter.com/muratkarslioglu/status/921072858628997121). The best part is that the users are independently evaluating OpenEBS and finding ways to automate their Compute, Network, Storage, and Data related Operations.\n\n![Towards Kubernetes](https://cdn-images-1.medium.com/max/800/0*XilwHl_ucs5K5fcK.jpg)\n\nToday, I can look back on our decision to use Kubernetes as a framework to build OpenEBS is one of the best decisions we have made. While the core of the Kubernetes community is helping DevOps teams treat Compute and Network as Code — we at OpenEBS are focused at extending Kubernetes to enable treating Storage and Data also as Code.\n\nI am delighted to announce that OpenEBS 0.5 is released with 300+ PRs coming from 50+ new community contributors, with several new features and bug fixes. Summary of changes are available in the [Release Notes](https://github.com/openebs/openebs/releases/tag/v0.5.0).\n\nSome notable changes include:\n\n- Storage Policy Enforcement Framework that allows DevOps teams to deploy a customized storage\n- Extend OpenEBS API Server to expose volume snapshot API\n- Support for deploying OpenEBS via helm charts\n- Support for monitor and get insights into OpenEBS volumes via Prometheus and Grafana\n- Sample Deployment YAMLs and corresponding Policy enabled Storage Classes for several stateful applications\n- Sample Deployment YAMLs for launching Kubernetes Dashboard for a preview of the changes done by OpenEBS Team to Kubernetes Dashboard\n\n***My favorite capability is the Storage Policy Framework that will enable each DevOps team to have their own storage controller — with their own storage policies. And the possibilities it will open-up!***\n\nImagine as a Developer I want to test my service against MySQL database with different datasets. On my staging PV (mysqldata), I can create multiple snapshots containing different datasets — say snaps like *sandy* and *wendy*. Now I can extend OpenEBS to support a new policy *\"openebs.io/jiva-replica-snap\"*, that can launch a new volume using snapshot for seed data.\n\nThe policy can be defined for snap — *sandy* as follows:\n\n # Define a SC referring to snapshot sandy\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mysvc-mysqldata-kiran-ds-sandy\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/jiva-replica-count: \"1\"\n openebs.io/jiva-source-pv: \"mysqldata\" \n openebs.io/jiva-replica-snap: \"mysqldata-ds-sandy\"\n\nand another one for *wendy*:\n\n # Define a storage classes supported by OpenEBS\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mysvc-mysqldata-kiran-ds-wendy\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/jiva-replica-count: \"1\"\n openebs.io/jiva-source-pv: \"mysqldata\"\n openebs.io/jiva-replica-snap: \"mysqldata-ds-wendy\"\n\nIn my PVC, I can now use the above StorageClasses (augmented with OpenEBS Storage Policies) to point to the different datasets and independently test my service.\n\nAll this from ***kubectl***.\n\nIn the background, OpenEBS will create a separate Containerized Storage Engine/Controller for my test database — by sourcing the data from the specified snapshot.\n\nAnd btw, we are on the [CNCF LandScape](https://github.com/cncf/landscape) under the Cloud Native Storage options, and decidedly leading the niche market for Containerized Storage Controller. You will hear more on this in the coming days at the KubeCon, Austin.\n\n![CNCF](https://cdn-images-1.medium.com/max/800/1*rdKFLGyf0hRDB_zcgGlywA.png)\n\nYou will hear more on OpenEBS 0.5 in the coming days at `KubeCon 2017`, Austin, Texas.\n\nI will be at the OpenEBS booths with my team and look forward to catching up with some of you in person.\n","notHasFeatureImage":true,"slug":"openebs-0.5-enables-storage-policies-for-kubernetes-persistent-volumes"},{"id":110,"title":"Storage Policies — It’s different this time","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"29-11-2017","tags":["Container","DevOps","Kubernetes","Storage Policy","Updates"],"excerpt":"One of the most common disbelief at the operator’s end would be the reports of an application’s (that consumed this storage) sudden death after introducing a much awaited shiny new storage feature.","content":"\n\n### Need for Storage Policy\n\nOne of the most common disbelief at the operator’s end would be the reports of an application’s (*that consumed this storage*) sudden death after introducing a much awaited shiny new storage feature. To make things worse the same surprise would be reciprocated from the storage provider as well. Neither the operator nor the storage fellow have any clues to this sudden strangeness.\n\nHaving gone through many such cycles of frustrations building storage features that satisfy the needs of every operator as well does not break stuff that are outside its control; I knew there was clearly a gap in storage design that needs to be talked about openly and bridged. In this article, I present storage policy as the solution to this pressing problem.\n\nThe moment I think about conceptualizing storage policy, my mind tries to battle against many odds. *Is this not what everyone (*read storage vendors*) has accomplished?*\n\n**Think again!!!** The chances are, it works for specific storage **“A”** and needs a radically different approach for the other storage **“B”**; assuming both of these provide the infrastructure for the operator’s storage needs. So the operator goes ahead and does some plumbings if lucky enough to get these infrastructures as white-box components. *At the end of the day, the operator is ready with a new bespoke program and feels so lucky*. Whatsoever, this does not last long due to the very behaviour of infra components. *This operator needs to be ready all the time for the next upgrade to above crafted program, **since little can be expected from the monolith storage fellas*. *Poor operator is left to fend for self, building all possible defence mechanisms*.\n\nGetting back to the need for a storage policy; *what is the edge in this incarnation of storage policy then*? Well, it is all about capitalising the essential features that being native to cloud and being operated from within a container provides *(I can only hope and pray for those storage controllers who cannot operate from the cloud(s) or can not be run from within the container(s))*.\n\nHaving said that, there is no innate need to design storage policy that is drastically different just to prove its greatness. Instead, there is a need to design, build, and apply policies that are agnostic to a storage implementation and is conducive to storage operations that are never ever meant to be interrupted.\n\n1. A Storage Policy should be simple to setup on Day 0\n2. It should be simple to expect the obvious on Day 1\n3. It should be simple enough to build an update whenever desired.\n\n### Is it that simple?\n\nAll this time, I have been describing the storage policy and its operations as simple. *Well, they can be described as simple, but not easy*. What was your feeling when you created one last time? *Have you ever felt creating a storage snapshot policy is easy?* How about a policy that does periodic restore\nof some randomly picked snapshots and marks them as *PASSED* or *FAILED* after verifying its data integrity. How did that failure alert you? Was it a slack notification? To make it further interesting, can you think of a snapshot policy that hooks into its consuming application’s life cycle events before trying out the snapshot (*Yes!!! I am suggesting those ‘freeze’ and ‘thaw’ exploits before and after a snapshot*).\n\n**Policies can be described as simple, but not easy.**\n\nRemember, these are not the only policies that can be thought of with respect to storage (*we have not even scratched the storage surface*). While all of these are achievable in a cloud and container native way, they may not be termed as easy.\n\nApplying storage policies is like a dangerous yet interesting sport. Like rock climbing, building a suitable policy can be learnt and then adapted based on the needs, workloads, environment, and so on. However, these policies if applied incorrectly can create a ripple-effect that can lead to increased costs and SLA misses which in turn leads to more support personnel on duty, and so on. As we read further, this new design of storage policies helps in eliminating the aforementioned impacts.\n\n### Bring your own YAML\n\nDid I just say YAML? Yes, I did and I shall explain it in a moment. Let us first explore the possible design angles of a storage policy. *Will it not be great to design a policy that fits well within an enterprise’s existing processes, its employees, its culture, its tools?* Is this too much to ask for?\n\nNo!!! *These are no more a set of good to have features but a must have checklist.* I believe expressing storage policy intents as YAMLs can pass all possible checks in an enterprise’s checklist. These current generation declarative intents are now fluent enough to be understood by APIs. In addition, one of YAML’s greatest strengths is its ability to abstract the entire logic that most of us understand as programmable code. This code now becomes truly yours since you have full control of these YAMLs. In other words, it is the operators who have the controlling rights.\n\nTo reiterate, there are few but really solid facts that makes this approach towards policy design a much better one than that of all its predecessors.\n\n***Here are my bets that makes this design different:***\n\n***Fact #1*** — To put things into perspective, the unassuming reader needs to look at the storage policies along with the current trends in cloud infrastructure as well as the trends in container engine. The cloud has of late become seamless. Thanks to Kubernetes which has been continuously bridging the impedance mismatch between different cloud providers. In addition, the communities involved in Container Storage Initiative, Container Runtime Interface, etc. are making intents as first class citizens. These declarative code pieces are thought of in a bottoms-up approach in each of these implementations which are then placed together coherently by the likes of Kubernetes. Now this is what is definitely more effective than just a tool that parses YAML and runs in isolation.\n\n***Fact #2*** — These intents are precise and parsed with appropriate validation checks to state with decisive control on the exact outcome. In other words they control the actual execution logic in addition to accepting input values via its declarative specifications. This grounds-up approach coupled with the nativity towards cloud as well as container engine is more suited to design storage policies that align with modern day DevOps’ practices.\n\n***Fact #3*** — Let us not forget the containers and the critical role they play in this age of cloud and orchestrators. Perhaps containers have become so ubiquitous and hence are easy to miss. I truly feel there are umpteen number of cool things that are yet to be discovered when we run a storage controller inside a container.\n\n**Fact #4** — All these also mean the intents that were once the sole prerogative of humans can now be built and operated by machines as well. This too with the same ease that the humans used to enjoy. After all, the ingredients *(#1, #2 and #3)* to make this possibility are all in place. This has come of age and is really an advancement in my opinion.\n\n***Bringing your own YAML really means setting your own policies and having complete control over their execution as well.***\n\n### Use the tool(s) you always loved\n\nA policy alone cannot justify its existence unless it is complimented with simple tools and automated processes. The careful reader might have already guessed it. Yes, I am talking of aligning storage policies with DevOps to realize its full potential.\n\n*Making storage policies more visible, more obvious, and enabling them to the enterprise’s established processes will make them simple to be believed and instil the faith to operate*. *Once again, there is a learning involved but the curve is not steep*.\n\nThese policies should offer the finest levels of control to the operators’ tools, their bots, and of course operators themselves when such a need arises. *For example finer granularity is craved for during rollbacks, automated downgrades, or blue-green deployments which are not uncommon in the world of storage infrastructures.*\n\nTo complete the DevOps cycle, these policies which can be handcrafted or system-generated can be submitted to the approver(s) (*which again can be a combination of humans as well as their loved tools*) as Pull Requests before being installed and applied against the storage.\n\n### Storage was the proverbial “Missing Cog”\n\nWe are seeing users finally achieving what many have dreamt of for so long  —  ***storage****(and the rest of the infrastructure)* truly being driven by the needs of the application and in a way that remains understandable and for a variety of reasons *(take infra as code for example)* trusted by humans. And now increasingly we see the recognition that containerised storage itself is another important ingredient.\n\nThanks to [OpenEBS](http://openebs.io) and more broadly containerised storage. *For the first time, every team and workload can have its **own fully functional storage controller**, with capabilities that have always been required by enterprise storage systems and that are still useful in taking care of *stateful workloads*. Our users do not want to give up the tools they used for the care and feeding of MySQL for example just because it now runs in a container. *This is possible by enabling capabilities like snapshots, versioning, encryption, and more **as knobs** to be able to be turned on/off for each workload.*\n\nThis incarnation of storage policies make it easy for many procedures for these workloads to be recorded as YAML; *the run book is truly code and so can easily be shared, versioned, and executed without humans having to play a role in the ugly details of managing storage system **A** or **B** or even **C**. Storage fades into the background. Time is ripe for the operators to rule.*\n","notHasFeatureImage":true,"slug":"storage-policies-—-it’s-different-this-time"},{"id":111,"title":"How to deploy a Cassandra Cluster/Ring on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"25-11-2017","tags":["Cassandra","Kubernetes","NoSQL","Solutions"],"excerpt":"Apache Cassandra is a distributed key-value store intended to run in a data center and also across multiple data centers. Initially it was designed as Facebook as an infrastructure for their messaging platform.","content":"\n\nApache Cassandra is a **distributed key-value store** intended to run in a data center and also across multiple data centers. Initially, it was designed as Facebook as an infrastructure for their messaging platform. Later it is open sourced, and today it’s one of the most active Apache projects.\nIf you are using eBay, Twitter, Spotify, or Netflix you are consuming data provided by Cassandra. For example, Netflix uses Cassandra to keep track of your current place in a streaming video, as well as movie ratings, bookmarks, and user history of 90+ million users.\nAmazing to see how much of this technology we consume in our day-to-day life. The feature that allowed me and my wife to start watching Stranger Things on our long trip on a tablet and continued on TV was depending on Cassandra. To give you an idea of its size, according to a [recent presentation](https://www.youtube.com/watch?v=2l0_onmQsPI), Cassandra serving Netflix has 250+ Clusters, 10,000+ Nodes, and 3+ PB of data.\nIn summary, Cassandra solves the problem of mapping the key-value pair to a server/node, in our case to a container. This mapping is called the **partitioner**. There are two common placement strategies used by Cassandra: **SimpleStrategy** or **NetworkTopologyStrategy**. SimpleStrategy uses partitioner Murmur3Partitioner by default. Both **Murmur3Partitioner** and **RandomPartitioner** partitioners uniformly distribute data to nodes across the cluster. Read and write requests to the cluster are evenly distributed while using these partitioners. Load balancing is simplified as each part of the hash range receives an equal number of rows on average. **Byte-Order Partitioner **is not recommended other than key range queries.\n\nFor development work, the SimpleStrategy class is acceptable. For production work, the NetworkTopologyStrategy class must be set. In production, you will end up with multiple rings using mostly NetworkTopology placement which is by itself extremely complex to plan.\nIf you want to learn the architecture of Cassandra, the University of Illinois has a great course on [Cloud Computing Concepts](https://www.coursera.org/learn/cloud-computing/home/welcome) and [Key-Value Stores](https://www.coursera.org/learn/cloud-computing/home/week/4) which covers internals of Cassandra. You can also find more about custom SeedProvider and Snitches [here](https://github.com/kubernetes/kubernetes/issues/24286).\nCassandra doesn’t like shared storage, therefore use of NFS or GlusterFS not recommended for Cassandra rings. It’s also recommended to use SSD or NVMe, since it’s essential to have low latency random reads and good sequential writes at the same time. These kinds of requirements can be only satisfied with OpenEBS like local and persistent storage solutions.\nTo achieve the best fault tolerance with Cassandra, you need to have an excellent understanding of the [**snitch**](http://cassandra.apache.org/doc/latest/operating/snitch.html)and placement strategies. There is a big debate on whether if Cassandra or the storage should handle the placement of data. My suggestion would be to have a balanced approach and have both. OpenEBS can help you to place your persistent volumes across the datacenter, multiple cloud vendors, and fault domains against Cassandra replica failures. First, you can **avoid rebalancing** your cluster in case of a datacenter failure. Second, in case of node failures in a rack, you can bring up the same node from a snapshot and **minimize the time needed to rebalance**.\n\nI will use Cassandra custom Kubernetes SeedProvider that allows discovery within Kubernetes clusters as they join the cluster and deploy using `gcr.io/google-samples/cassandra:v11` image from Google’s container registry.\n\nLet’s deploy our first three replica Cassandra cluster on our existing AWS K8s cluster with OpenEBS. If you are using on local minikube or datacenter, you can keep the default **SimpleStrategy** and **Murmur3Partitioner** in `cassandra.yaml` file.\n\n#### **Prerequisites**\n\n**Software**\n\n- [Docker ](https://docs.docker.com/engine/installation/)installed\n- Kubernetes 1.8.3+ RBAC enabled cluster\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n- Cassandra 3.x\n\n**Cloud Provider**\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n#### **Deploy Cassandra Stateful with Persistent Storage in one Region**\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following steps to launch a Cassandra service with any number of nodes you like.\n\nBy using environment variables, you can change values that are inserted into `cassandra.yaml`. Default **endpoint_snitch** is set to SimpleSnitch. I will change the Snitch to Ec2Snitch and also increase the replicas from 3 to 4 later.\n\nBefore getting started, check the status of the cluster using the following command.\n\n kubectl get nodes\n\nOn my setup, I have one master and four worker nodes on AWS in the same US West (Oregon) region and availability zone (us-west-2a).\n\n ubuntu@ip-172–23–1–236:~$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n ip-172–23–1–225.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–236.us-west-2.compute.internal Ready master 21h v1.8.3\n ip-172–23–1–32.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–35.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–46.us-west-2.compute.internal Ready 21h v1.8.3\n\nDownload OpenEBS GitHub repo to your host, where sample YAML files are stored.\n\n git clone [https://github.com/openebs/openebs.git](https://github.com/openebs/openebs.git)\n\nFirst list predefined OpenEBS storage classes available to you.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get storageclasses\n NAME PROVISIONER\n default kubernetes.io/aws-ebs\n etcd-backup-gce-pd kubernetes.io/gce-pd\n gp2 (default) kubernetes.io/aws-ebs\n openebs-cassandra openebs.io/provisioner-iscsi\n openebs-es-data-sc openebs.io/provisioner-iscsi\n openebs-jupyter openebs.io/provisioner-iscsi\n openebs-kafka openebs.io/provisioner-iscsi\n openebs-mongodb openebs.io/provisioner-iscsi\n openebs-percona openebs.io/provisioner-iscsi\n openebs-redis openebs.io/provisioner-iscsi\n openebs-standalone openebs.io/provisioner-iscsi\n openebs-standard openebs.io/provisioner-iscsi\n openebs-zk openebs.io/provisioner-iscsi\n\nGo to `openebs/k8s/demo/cassandra/` folder and edit `cassandra-statefulset.yaml` file.\n\n vi cassandra-statefulset.yaml\n\nThis file should look like below. You can edit and specify number of replicas preferred and your own OpenEBS storage class before applying.\n\n apiVersion: apps/v1beta1\n kind: StatefulSet\n metadata:\n name: cassandra\n labels:\n app: cassandra\n spec:\n serviceName: cassandra\n replicas: 3\n selector:\n matchLabels:\n app: cassandra\n template:\n metadata:\n labels:\n app: cassandra\n spec:\n containers:\n — name: cassandra\n image: gcr.io/google-samples/cassandra:v11\n imagePullPolicy: Always\n ports:\n — containerPort: 7000\n name: intra-node\n — containerPort: 7001\n name: tls-intra-node\n — containerPort: 7199\n name: jmx\n — containerPort: 9042\n name: cql\n resources:\n limits:\n cpu: “500m”\n memory: 1Gi\n requests:\n cpu: “500m”\n memory: 1Gi\n securityContext:\n capabilities:\n add:\n — IPC_LOCK\n lifecycle:\n preStop:\n exec:\n command: [“/bin/sh”, “-c”, “PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done”]\n env:\n — name: MAX_HEAP_SIZE\n value: 512M\n — name: HEAP_NEWSIZE\n value: 100M\n — name: CASSANDRA_SEEDS\n value: “cassandra-0.cassandra.default.svc.cluster.local”\n — name: CASSANDRA_CLUSTER_NAME\n value: “K8Demo”\n — name: CASSANDRA_DC\n value: “DC1-K8Demo”\n — name: CASSANDRA_RACK\n value: “Rack1-K8Demo”\n — name: CASSANDRA_AUTO_BOOTSTRAP\n value: “false”\n — name: POD_IP\n valueFrom:\n fieldRef:\n fieldPath: status.podIP\n readinessProbe:\n exec:\n command:\n — /bin/bash\n — -c\n — /ready-probe.sh\n initialDelaySeconds: 15\n timeoutSeconds: 5\n # These volume mounts are persistent. They are like inline claims,\n # but not exactly because the names need to match exactly one of\n # the stateful pod volumes.\n volumeMounts:\n — name: cassandra-data\n mountPath: /cassandra_data\n volumeClaimTemplates:\n — metadata:\n name: cassandra-data\n annotations:\n volume.beta.kubernetes.io/storage-class: openebs-cassandra\n spec:\n accessModes: [ “ReadWriteOnce” ]\n resources:\n requests:\n storage: 5G\n\nNote: There are few parameters you may want to modify.\n\n`apiVersion: apps/v1beta2` API group and version is introduced 1.8 release.\n\n`replicas: 3`, I’m starting with 3 replicas and will increase later.\n\n`image: gcr.io/google-samples/cassandra:v12` is the latest image available at the time I've tested.\n\n`volume.beta.kubernetes.io/storage-class: openebs-cassandra` I’m using a predefined OpenEBS storage class. You can modify it separately.\n\n#### Create a Cassandra Headless Service\n\nTo be able to have a simple discovery of the Cassandra seed node we need to create a “headless” service. If you view the `cassandra-service.yaml`file, you will notice that `clusterIP` is set to None. This will allow us to use KubeDNS for the Pods to discover the IP address of the Cassandra seed.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cat cassandra-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n labels:\n app: cassandra\n name: cassandra\n spec:\n clusterIP: None\n ports:\n — port: 9042\n selector:\n app: cassandra\n\nNow apply `cassandra-service.yaml` file to create headless service.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl apply -f cassandra-service.yaml\n service “cassandra” created\n\n#### Create a Cassandra StatefulSet\n\nMost applications deployed on Kubernetes should be **cloud-native** and rely on external resources for their data and state. However, stateful application and databases like Cassandra require stateful sets and persistent volumes to ensure resiliency. In this case, OpenEBS will provide us our persistent volume.\n\nThe StatefulSet is responsible for creating the Pods. Run the following command to start our Cassandra replicas.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl apply -f cassandra-statefulset.yaml\n statefulset “cassandra” created\n\n#### Validate the StatefulSet\n\nCheck if your StatefulSet has deployed using the command below. Time may take around 4–5 minutes to complete.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 3 3 5m\n\nIf you don’t see all 3 replicas ready, you can check status of pods to watch progress. For example, I ran `kubectl get pods` after 2 minutes below. First node was ready and second was still creating. All three pods were ready after around 5 minutes.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n cassandra-0 1/1 Running 0 2m\n cassandra-1 0/1 ContainerCreating 0 12s\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 13m\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 13m\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 12s\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 0/1 ContainerCreating 0 12s\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 12s\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 2m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 2m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0\n\nVerify that all the OpenEBS persistent volumes are created, the Cassandra headless service and replicas are running.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods -o wide\n NAME READY STATUS RESTARTS AGE IP NODE\n cassandra-0 1/1 Running 0 6m 10.2.2.4 ip-172–23–1–32.us-west-2.compute.internal\n cassandra-1 1/1 Running 0 4m 10.2.4.6 ip-172–23–1–46.us-west-2.compute.internal\n cassandra-2 1/1 Running 0 2m 10.2.1.6 ip-172–23–1–35.us-west-2.compute.internal\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 17m 10.2.4.3 ip-172–23–1–46.us-west-2.compute.internal\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 17m 10.2.3.2 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 4m 10.2.3.4 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 1/1 Running 0 4m 10.2.4.5 ip-172–23–1–46.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 4m 10.2.3.5 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-9c4bfcd6–4ss2r 1/1 Running 0 2m 10.2.1.4 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45–7tss8 1/1 Running 0 2m 10.2.3.6 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45-vfkrn 1/1 Running 0 2m 10.2.1.5 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 6m 10.2.1.3 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 6m 10.2.3.3 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0 6m 10.2.4.4 ip-172–23–1–46.us-west-2.compute.internal\n\nOn the list of the Pods above, you see 3 Pods running. Your Pod names should be cassandra-0, cassandra-1, cassandra-2 and the next pods would follow the ordinal number (cassandra-3, cassandra-4,...). Use this command to view the Pods created by the StatefulSet:\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get svc\n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n cassandra ClusterIP None 9042/TCP 7m\n kubernetes ClusterIP 10.3.0.1 443/TCP 22h\n maya-apiserver-service ClusterIP 10.3.0.204 5656/TCP 18m\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.188 3260/TCP,9501/TCP 5m\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.23 3260/TCP,9501/TCP 3m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.187 3260/TCP,9501/TCP 7m\n\n#### Verifying Successful Cassandra Deployment\n\nCheck if the Cassandra nodes are up, perform a **`nodetool status`** on cassandra-0 node :\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl exec -ti cassandra-0 — nodetool status\n Datacenter: DC1-K8Demo\n ======================\n Status=Up/Down\n |/ State=Normal/Leaving/Joining/Moving\n — Address Load Tokens Owns (effective) Host ID Rack\n UN 10.2.4.6 83.17 KiB 32 75.6% 3c93c7b7–61a7–4cf1-a407-cb47b1de0763 Rack1-K8Demo\n UN 10.2.1.6 65.65 KiB 32 59.5% 552569fe-c6df-4edb-a553–9efdcf682fb3 Rack1-K8Demo\n UN 10.2.2.4 83.12 KiB 32 64.9% 92060271-d8cd-48be-a489-c830a8553462 Rack1-K8Demo\n\nUN means node is **up** and in **normal** state. You will also notice that each node has 32 tokens. This is the default value, in production workloads, a good default value for this is 256. See more information [here](http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeVnodesUsing_c.html).\n\nThe **Owns** column suggests the data distribution percentage for the content placed into the Cassandra keyspaces.\n\n#### Create a Test Keyspace with Tables\n\nIdentify the IP Address of any of the Cassandra replicas, for example, Cassandra-0. This is available from the output of the `nodetool status` command executed above (10.2.4.6, 10.2.1.6, 10.2.2.4).\n\n**Cqlsh** is a Python-based utility that enables you to execute Cassandra Query Language (CQL). **CQL** is a declarative language that allows users to query Cassandra using semantics similar to SQL.\n\nInstall the python-minimal and python-pip apt packages and perform a pip install of Csqlsh using the following commands.\n\n sudo apt-get install -y python-minimal python-pip\n pip install cqlsh\n\nLogin to the CQL shell using the Cqlsh utility using the following command.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cqlsh 10.2.4.6 9042 — cqlversion=”3.4.2\"\n Connected to K8Demo at 10.2.4.6:9042.\n [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]\n Use HELP for help.\n cqlsh>\n\nAs I mentioned earlier, you have two placement options while creating a keyspace. You can either use SimpleStrategy or NetworkTopologyStrategy.\n\nYou can create a keyspace using SimpleStrategy with replication factor 2 by running the following commands.\n\n cqlsh> create keyspace ssks with replication = { ‘class’ : ‘SimpleStrategy’ , ‘replication_factor’ : 2 };\n\nI will create a keyspace using NetworkTopologyStrategy by running the following commands.\n\n cqlsh> create keyspace ntsks with replication = { ‘class’ : NetworkTopologyStrategy’, ‘DC1-K8Demo’ : 1 };\n\n cqlsh> describe keyspaces;\n\n ntsks system_schema system_auth system system_distributed system_traces\n\nTo use NetworkTopologyStrategy with data centers in a production environment, you need to change the default snitch, **SimpleSnitch** to a network-aware **Ec2Snitch**. You need to define one or more data center names in the snitch properties file, and use those data center names to define the keyspace; otherwise, Cassandra will fail to find a node. You can find the instructions to change the default Snitch [here](https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_switch_snitch.html).\n\nCreate a table with test content and view the data using the following commands.\n\n cqlsh> use ntsks;\n\n cqlsh:ntsks> create table inventory (id uuid,Username text,Email text,Age int,PRIMARY KEY ((id), Username));\n\n cqlsh:ntsks> insert into inventory (id, Username, Email, Age) values (1234b130-ae79–11e4-ab27–0700500c9a24, ‘Murat’, ‘murat@cloudbyte.com’, 40);\n\n cqlsh:ntsks> select * from inventory;\n\n id | username | age | email\n — — — — — — — — — — — — — — — — — — — + — — — — — + — — -+ — — — — — — — — — — -\n 1234b130-ae79–11e4-ab27–0700500c9a24 | Murat | 37 | murat@cloudbyte.com\n\n (1 rows)\n\nFlush the data to ensure it is written to a disk from the memtable (memory) using the following command.\n\n kubectl exec cassandra-0 — nodetool flush ntsks\n\n#### Delete the Test Keyspace\n\nVerify the masterless nature of Cassandra StatefulSet by deleting the keyspace from another replica, in this example, Cassandra-1.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cqlsh 10.2.1.6 9042 — cqlversion=”3.4.2\"\n Connected to K8Demo at 10.2.1.6:9042.\n [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]\n Use HELP for help.\n cqlsh> use ntsks;\n\n cqlsh:ssks> select * from Inventory;\n\n id | username | age | email\n — — — — — — — — — — — — — — — — — — — + — — — — — + — — -+ — — — — — — — — — — -\n 1234b130-ae79–11e4-ab27–0700500c9a24 | Murat | 37 | murat@cloudbyte.com\n\n (1 rows)\n\n cqlsh> drop keyspace ntsks;\n\nVerify that the keyspace is deleted successfully using the following command.\n\n cqlsh> describe keyspaces\n\n system_traces system_schema system_auth system system_distributed\n\n#### Scale the StatefulSet\n\nTo increase or decrease the size of your StatefulSet you can use the scale command:\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl scale — replicas=4 statefulset/cassandra\n statefulset “cassandra” scaled\n\nWait a minute or two and check if it worked:\n\n $ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 4 4 1h\n\nIf you watch the Cassandra pods deploy, they should be created sequentially.\n\nYou can view the list of the Pods again to confirm that your Pods are up and running.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 4 4 1h\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods -o wide\n NAME READY STATUS RESTARTS AGE IP NODE\n cassandra-0 1/1 Running 0 1h 10.2.2.4 ip-172–23–1–32.us-west-2.compute.internal\n cassandra-1 1/1 Running 0 1h 10.2.4.6 ip-172–23–1–46.us-west-2.compute.internal\n cassandra-2 1/1 Running 0 1h 10.2.1.6 ip-172–23–1–35.us-west-2.compute.internal\n cassandra-3 0/1 Running 0 1m 10.2.3.9 ip-172–23–1–225.us-west-2.compute.internal\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 1h 10.2.4.3 ip-172–23–1–46.us-west-2.compute.internal\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 1h 10.2.3.2 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 1h 10.2.3.4 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 1/1 Running 0 1h 10.2.4.5 ip-172–23–1–46.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 1h 10.2.3.5 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-9c4bfcd6–4ss2r 1/1 Running 0 1h 10.2.1.4 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45–7tss8 1/1 Running 0 1h 10.2.3.6 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45-vfkrn 1/1 Running 0 1h 10.2.1.5 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-ctrl-5b6d99869–7gxv5 1/1 Running 0 1m 10.2.3.7 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-rep-5fc8b95cd-6vfbt 1/1 Running 0 1m 10.2.2.5 ip-172–23–1–32.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-rep-5fc8b95cd-h22qz 1/1 Running 0 1m 10.2.3.8 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 1h 10.2.1.3 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 1h 10.2.3.3 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0 1h 10.2.4.4 ip-172–23–1–46.us-west-2.compute.internal\n\nYou can perform a `nodetool status` to check if the other Cassandra nodes have joined and formed a Cassandra cluster.\n\n $ kubectl exec -ti cassandra-0 — nodetool status\n Datacenter: DC1-K8Demo\n ======================\n Status=Up/Down\n |/ State=Normal/Leaving/Joining/Moving\n — Address Load Tokens Owns (effective) Host ID Rack\n UN 10.2.4.6 174.97 KiB 32 59.7% 3c93c7b7–61a7–4cf1-a407-cb47b1de0763 Rack1-K8Demo\n UN 10.2.1.6 182.32 KiB 32 43.8% 552569fe-c6df-4edb-a553–9efdcf682fb3 Rack1-K8Demo\n UN 10.2.2.4 169.7 KiB 32 42.5% 92060271-d8cd-48be-a489-c830a8553462 Rack1-K8Demo\n UN 10.2.3.9 90.81 KiB 32 54.1% 47e8c9e2-a6d9–4276–88ae-6fe2256ca2af Rack1-K8Demo\n\nYou will need to wait for the status of the nodes to be Up and Normal (UN) to execute the commands in the next steps.\n\n#### Troubleshooting\n\nIf your Cassandra instance is not running properly, you may check the logs using the command below. Replace with your pod name. For example, `cassandra-0`:\n\n kubectl logs \n\nIf your Cassandra nodes are not joining, delete your controller/statefulset then delete your Cassandra service:\n\n kubectl delete statefulset cassandra\n\nIf you created the Cassandra StatefulSet:\n\n kubectl delete svc cassandra\n\nTo delete everything:\n\n kubectl delete statefulset,pvc,pv,svc -l app=cassandra\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-deploy-a-cassandra-cluster-ring-on-kubernetes-openebs/)*.\n","notHasFeatureImage":false,"slug":"how-to-deploy-a-cassandra-cluster/ring-on-kubernetes-+-openebs"},{"id":112,"title":"How to install OpenEBS on IBM Cloud Private","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"19-11-2017","tags":["Helm Charts","OpenEBS","Solutions","ICP"],"excerpt":"What is IBM Cloud Private? IBM Cloud Private (ICP) is a new application platform that is based on Kubernetes and provides services for developing and managing on-premises containerized applications. ","content":"\n## What is IBM Cloud Private?\n\n**IBM Cloud Private (ICP)** is a new application platform that is based on **Kubernetes** and provides services for developing and managing **on-premises** containerized applications. ICP Community Edition *(ICP-CE)* is distributed free of charge for non-production use and is available on Docker Hub. For commercial use, you would need the Enterprise package.\n\nIn my previous blog post, [Introduction to IBM Cloud Private](http://containerized.me/introduction-to-ibm-cloud-private/), I have covered step-by-step installation of ICP 2.1. This time I will focus on configuring **OpenEBS** as a **persistent storage** option and deploying a stateful workload (MongoDB) using OpenEBS storage classes.\n\n## Prerequisites\n\n### Hardware\n\n- Minimum three x64 servers\n\n### Software\n\n- [Ubuntu Server 16.04 LTS](https://www.ubuntu.com/download/server)\n- IBM Cloud Private 2.1\n- [OpenEBS](https://github.com/openebs/openebs)\n\n### Install IBM Cloud Private\n\nFollow instructions from [Introduction to IBM Cloud Private](http://containerized.me/introduction-to-ibm-cloud-private/) to deploy a multi-node ICP cluster.\n\n### Install OpenEBS on ICP\n\n1. Log in to the ICP console and go to the **Admin/Repositories** menu.\n\n![OpenEBS on ICP](https://cdn-images-1.medium.com/max/800/0*PPZPNSr9_mW_9AZq.png)\n\n1. Click **Add repository**.\n\n![Add repository](https://cdn-images-1.medium.com/max/800/0*ZNaLIkk1gxFLWUJK.png)\n\n1. Add a chart repository with the following parameters:\n — **Name:** openebs-charts\n — **URL:** [https://openebs.github.io/charts/](https://openebs.github.io/charts/)\n\n![Confirm charts](https://cdn-images-1.medium.com/max/800/0*2m2J6V9YhnYk5_Cx.png)\n\n1. After you click **Add**, confirm that **openebs-charts** is listed under Repositories.\n\n![Catalog menu](https://cdn-images-1.medium.com/max/800/0*wkPxIB_Q2DevkgWh.png)\n\n1. Go to the **Catalog** menu, select **openebs** from the list.\n\n![Configure](https://cdn-images-1.medium.com/max/800/0*7Lt6IE4f_da0jZEB.png)\n\n1. On OpenEBS chart instructions page, click **Configure**.\n2. Configure OpenEBS deployment with the following parameters:\n — **Release name:** openebs- (you need to pick a unique name)\n — **Target Namespace:** default (namespace should be the same as your workload)\n — **rbacEnable:** true\n — **image pullPolicy:** IfNotPresent\n — **apiserver image:** openebs/m-apiserver\n — **apiserver tag:** 0.4.0\n — **provisione image:** openebs/openebs-k8s-provisioner\n — **provisioner tag:** 0.4.0\n — **jiva image:** openebs/jiva:0.4.0\n — **replicas:** 2 (Number of Jiva Volume replicas)\n\n![Installation](https://cdn-images-1.medium.com/max/800/0*qfLs4pg_3TE1PbCB.png)\n\n1. Click **Install**. When finished click **View Helm Release.**\n\n![Storage classes](https://cdn-images-1.medium.com/max/800/0*raLyHiJeZ0hC_BAk.png)\n\n1. On the Helm Release page, you can see the status of OpenEBS, deployment, and available **Storage Classes**.\n\n![Deploy stateful application](https://cdn-images-1.medium.com/max/800/0*-gCAd374s2jXY3AP.jpg)\n\n1. Now, let’s try to deploy a stateful app on OpenEBS.\n\n### Install MongoDB on OpenEBS\n\n1. Under **Catalog**, select **ibm-mongodb-dev** and click **Configure**.\n2. Configure MongoDB deployment with the following parameters:\n — **Release name:** mongodb- (you need to pick a unique name here)\n — **Target Namespace:** default (same as OpenEBS)\n — **persistence enabled:** true\n — **persistence useDynamicProvisioning:** true\n — **dataVolume storageClassName:** openebs-mongodb\n — **dataVolume size:** 2G (default is 20Gi, remove “i” — in current version it is not supported)\n — **database password:** mongo\n Accept the license agreements, keep all the other values as default and click **Install**.\n\n![Workloads release](https://cdn-images-1.medium.com/max/800/0*UTiLWk3zOy5bw_Wh.png)\n\n1. Go to **Workloads/Helm Releases** and select your MongoDB release. Under the **PersistentVolumeClaim** table you are going to see the volume claim and OpenEBS storage class.\n\n![Workloads deployment](https://cdn-images-1.medium.com/max/800/0*PNNp0nDxsZXzYwIH.png)\n\n1. If you go to the **Workloads/Deployments** page, you can find the storage controller and two volume replicas (as configured) running.\n\n![Repository](https://cdn-images-1.medium.com/max/800/0*uaEIPO8n2vY0yUet.png)\n\n1. Confirm that replicas are running on separate nodes. Click on the PVC name ending with **rep** (Example:pvc-23025190-c516–11e7-b45e-e8fd90000064-rep). Scroll down, and you will see that pods are running on separate hosts.\n\n![Deployment successful](https://cdn-images-1.medium.com/max/800/0*pD7rHAX_D8_cxcfl.png)\n\nYou have successfully deployed a stateful application on a persistent block storage presented by OpenEBS.\n\n### How does storage HA work for stateful workloads?\n\nHigh Availability storage (HA storage) is a storage system that is continuously operational. Redundancy is the key feature of HA storage as it allows data to be kept in more than one place while ensuring data protection and consistency.\n\nAn **OpenEBS Jiva Volume** is a controller deployed during the OpenEBS installation. Volume replicas are defined by the parameter we set above. The controller is an **iSCSI target** while the replicas play the role of a disk. The controller exposes the iSCSI target while the actual data is written. The controller and each replica run inside a dedicated container.\n\nAn OpenEBS Jiva Volume controller exists as a single instance, but there can be multiple instances of OpenEBS Jiva volume replicas. Persistent data is synchronized between replicas.\n\nOpenEBS Jiva Volume HA is based on various scenarios as explained in the following sections.\n\nNOTE: Each replica is scheduled in a unique K8s node, and a K8s node never has two replicas of one OpenEBS volume.\n\n### What happens when an OpenEBS volume controller pod crashes?\n\nKubernetes automatically re-schedules the controller as a new Kubernetes pod.\n Policies are in place that ensures faster rescheduling.\n\n### What happens when a K8s node that hosts OpenEBS volume controller goes offline?\n\nThe controller is automatically re-scheduled as a new Kubernetes pod.\n Policies are in place that ensures faster rescheduling.\n If Kubernetes node is unavailable, the controller gets scheduled on one of the available nodes.\n\n### What happens when an OpenEBS volume replica pod crashes for reasons other than node not-ready and node unreachable?\n\nThe replica is autoamtically re-scheduled as a new Kubernetes pod.\n The replica may or may not be re-scheduled on the same K8s node.\n There is data loss with this newly scheduled replica if it gets re-scheduled on a different K8s node.\n\n### What happens when a K8s node that hosts OpenEBS volume replica goes offline?\n\nThere is no storage downtime as the other available replica displays inputs/outputs.\n Policies are in place that does not allow re-scheduling of crashed replica (as the replica is tied to a node’s resources) to any other node.\n\n---\n\n*Originally published at *[*Containerized Me*](http://containerized.me/how-to-install-openebs-on-ibm-cloud-private/)*.*\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-ibm-cloud-private"},{"id":113,"title":"How to deploy Jenkins on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"16-11-2017","tags":["Jenkins","Kubernetes","Open Source","Solutions","OpenEBS"],"excerpt":"Modern development requires Continuous Integration / Continuous Delivery (CI/CD) and it means building and validating your software on every commit to make sure your development & test environments are always up-to-date. ","content":"\nModern development requires [Continuous Integration](https://aws.amazon.com/devops/continuous-integration/) / [Continuous Delivery](https://aws.amazon.com/devops/continuous-delivery/) (**CI/CD**) and it means building and validating your software on every commit to make sure your development & test environments are always up-to-date. This level of automation is a combination of cultural philosophies (aka **DevOps**) and practices. CI/CD increases an organization’s ability to deliver applications and services at high velocity. Jenkins serves as the **workflow engine** to manage this **CI/CD pipeline** from source to delivery.\n\nDeploying Jenkins on Kubernetes provides the following benefits:\n\n- Isolates different jobs from one another\n- Quickly clean a job’s workspace\n- Dynamically deploy or schedule jobs with Kubernetes pods\n- Allows increased resource utilization and efficiency\n- Dynamically scale up Jenkins slaves on demand\n\nEspecially, running dynamic slaves in a Kubernetes/Docker environment and automating the scaling of Jenkins slaves running in Kubernetes on top of OpenEBS can **minimize the deployment time and cost**. With OpenEBS, you can build extremely scalable test cycles. You will be able to create instant snapshots (thanks to the [**CoW**](https://en.wikipedia.org/wiki/Copy-on-write)) from the master and **deploy new slaves faster and dynamically on-demand**. This process will eliminate the need to perform container-to-container copies.\n\nIn Jenkins, slaves are optional. OpenEBS can also help when you have a smaller environment and running a **monolithic master**. In that model, state on the master would be lost when you shut down the Jenkins master service. When using monolithic master on OpenEBS, your volume is persistent and replicated over to n nodes *(defined in your OpenEBS storage class)*. In that case, the master can exit, even if your node fails it can start on other nodes, migrate from private to public cloud, vice-versa when needed and your data will follow you.\n\nLet’s deploy Jenkins on our existing K8s cluster with OpenEBS. You will notice that it’s not much different than deploying on local storage, except your data will be protected with OpenEBS.\n\n## Prerequisites\n\n### Software\n\n- [Docker](https://docs.docker.com/engine/installation/) installed\n- Kubernetes 1.7.3+ RBAC enabled cluster\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n## Deploy Jenkins Pod with Persistent Storage\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following simple steps to launch Jenkins service with a monolithic master.\n\nBefore getting started, check the status of the cluster using the following command.\n\n kubectl get nodes\n\nIn my environment, I have one master and two worker nodes.\n\n ubuntu@ip-172–23–1–115:~$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n ip-172–23–1–115.us-west-2.compute.internal Ready master 2h v1.8.3\n ip-172–23–1–144.us-west-2.compute.internal Ready 2h v1.8.3\n ip-172–23–1–244.us-west-2.compute.internal Ready 2h v1.8.3\n\nDownload the `Jenkins.yml` file to your host, which has access to kubectl.\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/jenkins/jenkins.yml\n\nThis file looks like below. You can edit and specify a different OpenEBS storage class before you apply.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: jenkins-claim\n annotations:\n volume.beta.kubernetes.io/storage-class: openebs-standard\n spec:\n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n — -\n apiVersion: extensions/v1beta1\n kind: Deployment\n metadata:\n name: jenkins\n spec:\n replicas: 1\n template:\n metadata:\n labels:\n app: jenkins-app\n spec:\n securityContext:\n fsGroup: 1000\n containers:\n — name: jenkins\n imagePullPolicy: IfNotPresent\n image: jenkins/jenkins:lts\n ports:\n — containerPort: 8080\n volumeMounts:\n — mountPath: /var/jenkins_home\n name: jenkins-home\n volumes:\n — name: jenkins-home\n persistentVolumeClaim:\n claimName: jenkins-claim\n — -\n apiVersion: v1\n kind: Service\n metadata:\n name: jenkins-svc\n spec:\n ports:\n — port: 80\n targetPort: 8080\n selector:\n app: jenkins-app\n type: NodePort\n\nNow apply `jenkins.yml` file.\n\n kubectl apply -f jenkins.yml\n\n![results](/images/blog/results.png)\n\nGet the status of running pods using the following command.\n\n kubectl get pods\n\nResult should like similar to below, and the Jenkins pod running.\n\n ubuntu@ip-172–23–1–115:~$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n jenkins-797b888448-pfx8x 1/1 Running 0 11m\n maya-apiserver-5994b58bbb-ck2tv 1/1 Running 0 2h\n openebs-provisioner-6f45dcf459-qjdlx 1/1 Running 0 2h\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-ctrl-864fcb6f74–2phfw 1/1 Running 0 11m\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-rep-575d85d96c-dk4dq 1/1 Running 0 11m\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-rep-575d85d96c-pzrgn 1/1 Running 0 11m\n\nAs you noticed, your OpenEBS controller `pvc-…-ctrl-…` and two copies of persistent volumes `pvc-…-rep-…` are also deployed and running.\n\nGet the status of underlying persistent volumes used by Jenkins deployment using the following command.\n\n kubectl get pvc\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n jenkins-claim Bound pvc-94586807-cb09–11e7-b125–064dff6dc2a2 5G RWO openebs-standard 22m\n\nGet the status of Jenkins service using the following command:\n\n kubectl get svc\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl get svc\n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n jenkins-svc NodePort 10.3.0.17 80:31705/TCP 25m\n kubernetes ClusterIP 10.3.0.1 443/TCP 3h\n maya-apiserver-service ClusterIP 10.3.0.34 5656/TCP 3h\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-ctrl-svc ClusterIP 10.3.0.100 3260/TCP,9501/TCP 25m\n\n### Launching Jenkins\n\nThe Jenkins deployment YAML file `jenkins.yaml` we have used above creates a NodePort service type to make Jenkins available outside the cluster.\n\nGet the node IP Address that is running the Jenkins pod using the following command.\n\nNote: Replace your pod name with your the pod name returned when you ran `kubectl get pods` command.\n\n kubectl describe pod jenkins-797b888448-pfx8x | grep Node:\n\nExample output below:\n\n kubectl describe pod jenkins-797b888448-pfx8x | grep Node:\n Node: ip-172–23–1–144.us-west-2.compute.internal/172.23.1.144\n\nGet the port number from the Jenkins service using the following command:\n\n kubectl describe svc jenkins-svc | grep NodePort:\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl describe svc jenkins-svc | grep NodePort:\n NodePort: 31705/TCP\n\nIP above is your private IP on AWS, which can be used if you are accessing through another instance on AWS. To access it remotely, you also need to open that port on E2C instance’s security group.\n\nGo to the Network & Security -> Security Group settings in the left hand navigation\nFind the **Security Group** that your instance is a part of. Click on **Inbound Rules**. Click on **Edit** and **Add Rule** button. Then add HTTP (port 31705). Click **Save**.\n\nNow, combine your public IP and port number and open that in your browser. In my case, it is [https://34.223.235.50:31705.](https://34.223.235.50:31705.)\n\nOnce you access the URL the Getting Started page is displayed. Follow the procedure below to setup Jenkins.\n\nProvide the [cci]initialAdminPassword[/cci] in the Unlock Jenkins screen and copy the password in the [cci]Administrator password[/cci] field. Click **Continue**.\n\n![unclock jenkins](/images/blog/unlock-jenkins.png)\n\nGet the password using the following command:\n\n kubectl exec -it jenkins-797b888448-pfx8x cat /var/jenkins_home/secrets/initialAdminPassword\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl exec -it jenkins-797b888448-pfx8x cat /var/jenkins_home/secrets/initialAdminPassword\n 5aa044d226d1466eb84621e75e369c64\n\nOn the **Customize Jenkins** screen click on **Install suggested plugins**.\n\n![customize jenkins](/images/blog/customize-jenkins.png)\n\nConfigure the Administrator user in the **Create First Admin User** screen. Fill in the following fields.\n\n**Username:** — Key in the administrator username.\n**Password:** — Key in the password for the administrator.\n**Confirm password:** — Key in the password again and confirm.\n**Full name:** — Key in the administrator’s full name.\n\nClick **Continue as admin** if you want to perform further administrator tasks or click **Save and Finish**.\n You can now start using Jenkins!\n\n![jenkins-is-ready](/images/blog/jenkins-is-ready.png)\n\n\n_Originally published at [*Containerized Me*](http://containerized.me/how-to-deploy-jenkins-on-kubernetes-openebs/)_.\n","notHasFeatureImage":false,"slug":"how-to-deploy-jenkins-on-kubernetes-+-openebs"},{"id":114,"title":"How to deploy a PostgreSQL Cluster on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"02-11-2017","tags":["Crunchy","Kubectl","Solutions","Kubernetes","OpenEBS"],"excerpt":"Why Postgres on Kubernetes? Well, the answer is in the question. If you are already running Kubernetes on some form of cloud, you understand the ease-of-use, scalability, and monitoring benefits of Kubernetes that you can apply to your database at scale.","content":"\n## Why Postgres on Kubernetes?\n\nWell, the answer is in the question. If you are already running Kubernetes on some form of cloud, you understand the **ease-of-use**, **scalability**, and **monitoring** benefits of Kubernetes that you can apply to your database at scale.\n\nPostgreSQL is the **preferred** relational database for most developers around, although setting up a highly available Postgres cluster from scratch is always a challenge, being **cloud-native** adds a bit to the difficulty.\n\nThere are many ways to run **high availability** with PostgreSQL; for a list, see the [PostgreSQL Documentation](https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling). To be honest, manually setting it up is quite painful, while there are better ways available. My favorite **cloud-native** Postgres cluster deployment projects are [Crunchy Data](https://www.crunchydata.com/)’s, [Sorint.lab](https://www.sorint.it/)’s [Stolon](https://github.com/sorintlab/stolon) and [Zalando](https://jobs.zalando.com/tech/)’s [Patroni](https://github.com/zalando/patroni)/[Spilo](https://github.com/zalando/spilo).\n\nSince availability requires multi-node Kubernetes deployment instead of local minikube setup, I’ll deploy crunchy-postgres on my existing K8s cluster on AWS with two worker nodes. If you don’t have a Kubernetes cluster yet, see the [instructions to deploy one using StackPointCloud](http://containerized.me/how-to-install-openebs-on-aws-using-stackpointcloud/). Instructions after that are the same in any cloud or on-premises deployment.\n\n## Prerequisites\n\n### Software\n\n- [crunchy-postgres](https://hub.docker.com/r/crunchydata/crunchy-postgres/) (for cluster deployment)\n- [Docker](https://docs.docker.com/engine/installation/)installed\n- Kubernetes 1.5+ cluster installed\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n### Deploy Crunchy PostgreSQL cluster using kubectl\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following simple steps to launch a highly available PostgreSQL service with one master and one replica.\n\nDownload the files to your host, which has access to kubectl\n\n cd $HOME\n git clone https://github.com/openebs/openebs.git\n cd openebs/k8s/demo/crunchy-postgres \n\n### Create the Stateful Set\n\nThe deployment will use the default images and credentials defined in the set.json file. To set custom users and passwords:\n\n vi ~/openebs/k8s/demo/crunchy-postgres/set.json\n\nJSON file should look like below, feel free to edit the number of replicas, credentials and storage capacity. Default uses the **openebs-standard** storage class, and it is 400M.\n\n {\n \"apiVersion\": \"apps/v1beta1\",\n \"kind\": \"StatefulSet\",\n \"metadata\": {\n \"name\": \"pgset\"\n },\n \"spec\": {\n \"serviceName\": \"pgset\",\n \"replicas\": 2,\n \"template\": {\n \"metadata\": {\n \"labels\": {\n \"app\": \"pgset\"\n }\n },\n \"spec\": {\n \"containers\": [\n {\n \"name\": \"pgset\",\n \"image\": \"crunchydata/crunchy-postgres:centos7–9.6–1.4.0\",\n \"ports\": [\n {\n \"containerPort\": 5432,\n \"name\": \"postgres\"\n }\n ],\n \"env\": [\n {\n \"name\": \"PG_MASTER_USER\",\n \"value\": \"master\"\n },\n {\n \"name\": \"PGHOST\",\n \"value\": \"/tmp\"\n },\n {\n \"name\": \"PG_MODE\",\n \"value\": \"master\"\n },\n {\n \"name\": \"PG_MASTER_PASSWORD\",\n \"value\": \"password\"\n },\n {\n \"name\": \"PG_USER\",\n \"value\": \"testuser\"\n },\n {\n \"name\": \"PG_PASSWORD\",\n \"value\": \"password\"\n },\n {\n \"name\": \"PG_DATABASE\",\n \"value\": \"userdb\"\n },\n {\n \"name\": \"PG_ROOT_PASSWORD\",\n \"value\": \"password\"\n }\n ],\n \"volumeMounts\": [\n {\n \"name\": \"pgdata\",\n \"mountPath\": \"/pgdata\",\n \"readOnly\": false\n }\n ]\n }\n ]\n }\n },\n \"volumeClaimTemplates\": [\n {\n \"metadata\": {\n \"name\": \"pgdata\"\n },\n \"spec\": {\n \"accessModes\": [\n \"ReadWriteOnce\"\n ],\n \"storageClassName\": \"openebs-standard\",\n \"resources\": {\n \"requests\": {\n \"storage\": \"400M\"\n }\n }\n }\n }\n ]\n }\n }\n\nSave the file and run the statefulset:\n\n ./run.sh\n\nThe above step will automatically create the OpenEBS volumes required for master and replica postgresql containers and few other Kubernetes objects:\n\n- Persistent Volumes (pvc-{UID1}, pvc-{UID2})\n- Persistent Volume Claim (pgdata-pgset-0, pgdata-pgset-1)\n- Replica Sets (pvc-{UID1}-ctrl-{random1},pvc-{UID1}-rep-{random2},pvc-{UID2}-ctrl-{random3},pvc-{UID4}-ctrl-{random4})\n- Service Account (pgset-sa)\n- Services (pgset, pgset-master, pgset-replica)\n- StatefulSet (pgset)\n- Pods (pgset-0, pgset-1)\n\n![GIF displaying OpenEBS volumes required for master and replica postgresql containers and few other Kubernetes objects](https://cdn-images-1.medium.com/max/800/0*_WTDmIAcGNUGL0zn.gif)\n\nThe volume details can be inspected using the standard kubectl commands. To check **persistent volume claims**:\n\n kubectl get pvc\n\n![Screenshot showing persistent volume claims](https://cdn-images-1.medium.com/max/800/0*Jj59F2CWdQqKOkjW.png)\n\nCheck **persistent volumes**:\n\n kubectl get pv\n\n![Screenshot showing persistent volumes](https://cdn-images-1.medium.com/max/800/0*cm0u7Ea_12FvQRC4.png)\n\nList the **services**, and you will see pgset, master and replica created:\n\n kubectl get service\n\n![Listing services](https://cdn-images-1.medium.com/max/800/0*d5PjsFswTOOSBAcq.png)\n\nList the **statefulsets**, and you will see pgset listed with two desired and current sets:\n\n![Listing statefulsets](https://cdn-images-1.medium.com/max/800/0*F3eKWl181xp3yKLJ.png)\n\nIf you use the **Kubernetes Dashboard**, you can see the same under **Workloads > Stateful Sets** and quickly scale up as well.\n\n![Kubernetes Dashboard](https://cdn-images-1.medium.com/max/800/0*fQO6h-cj00rbePIi.png)\n\n### Test your Database\n\nIf it is not installed previously, install psql client:\n\n sudo apt-get install postgresql-client\n\nTest the master as follows (default password is “password”, unless you changed it):\n\n psql -h pgset-master -U testuser password -c ‘table pg_stat_replication’\n\nAbove command should return output indicating that a single replica is connecting to the master.\n\nNow, test the replica as follows:\n\n psql -h pgset-replica -U testuser password -c ‘create table foo (id int)’\n\nThis command should fail as the replica is read-only within a PostgreSQL cluster.\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/)*.\n","notHasFeatureImage":false,"slug":"how-to-deploy-a-postgresql-cluster-on-kubernetes-+-openebs"},{"id":115,"title":"How to Install OpenEBS on AWS using StackPointCloud?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"23-10-2017","tags":["Kubernetes","Solutions","StackPointCloud","Ubuntu","OpenEBS"],"excerpt":"What is StackPointCloud? StackPointCloud is a managed Kubernetes control plane to build cloud-native stacks on AWS, Google Cloud (GKE & GCE), Azure & DigitalOcean. ","content":"\n## What is StackPointCloud?\n\nStackPointCloud is a managed Kubernetes control plane to build cloud-native stacks on AWS, Google Cloud (GKE & GCE), Azure & DigitalOcean. StackPointCloud simplifies installation and aggregation of multiple Kubernetes clusters pretty much on any platform. Even if you are an expert, provisioning your own Kubernetes stack their easy to use interface and capabilities to centralize all your deployments in one place is compelling. StackPointCloud is free for the first 30 days and $49.95 a month after for any number of Kubernetes clusters.\n\n## Prerequisites\n\nMinimum requirements for deploying your Kubernetes clusters on StackPointCloud:\n\n### Hardware\n\n- None\n\n### Software\n\n- [OpenEBS](https://github.com/openebs/openebs)\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account (Other major providers supported by StackPoint, but not covered in this article)\n\n## Start your StackPoint Trial\n\nFirst, go to [stackpoint.io](https://stackpoint.io/) and click on the **Launch a Cluster** button to start your free trial.\n\n![Launch a Cluster button in stackpoint.io](https://cdn-images-1.medium.com/max/800/0*3Iro4mlPVlQolQfh.png)\n\nThen choose your cloud provider. In this example, I will use **AWS**.\n\n![Configure Access to AWS](https://cdn-images-1.medium.com/max/800/0*s0vkUYR7sJXoR6IU.png)\n\n#### Configure Access to AWS\n\nOn the next screen, we need to configure our provider. You need to provide AWS Access Key ID and Secret Access Key and optionally your SSH Key.\n\n![Cloud providers](https://cdn-images-1.medium.com/max/800/0*_2SUsICymTDtGlwK.png)\n\nIf you don’t know where to find them, follow the instructions [here](https://stackpointcloud.com/community/tutorial/how-to-create-auth-credentials-on-amazon-web-services-aws) to create your user.\n\nClick on **Add Credentials** button.\n\n![Add Credentials](https://cdn-images-1.medium.com/max/800/0*5LX2XDbBqhnm1au8.png)\n\nAfter you add your credentials, click on **Submit**.\n\n## Configure K8s Cluster\n\nOn the “Configure your cluster” page click the edit button on **Distribution** and choose **Ubuntu 16.04 LTS**.\n\n![Configure K8s Cluster](https://cdn-images-1.medium.com/max/800/0*ty0IA_1uuDxaCQoX.png)\n\nChange the **Cluster Name** something meaningful like **OpenEBS Demo**.\n\n![Change the Cluster Name](https://cdn-images-1.medium.com/max/800/0*50cyzQI-2DZIX-AG.png)\n\nI could separate my etcd into 3 nodes dedicated cluster, but for a functional demo hosting it on the same cluster works perfectly fine. You can leave all other options as default. Now click on **Submit** to create your cluster. This should take around 5–8 minutes to bring up one Master and two Workers Kubernetes Cluster.\n\n## Import OpenEBS Helm Charts\n\nClick on the **Solutions** tab on the top of the screen and select **Import Charts** from the upper left.\n\n![Import OpenEBS Helm Charts](https://cdn-images-1.medium.com/max/800/0*vZr9hqN35SCCsx-a.png)\n\nAdd the chart repo with the following details: \n — **name :** openebs-charts \n — **type :** packaged-charts \n — **repo url : **[https://openebs.github.io/charts/](https://openebs.github.io/charts/)\n\nClick on **Review Repository**.\n\n![Update chart repo](https://cdn-images-1.medium.com/max/800/0*lkT38CLmsESK2i1T.png)\n\nMake sure **Access Verified** shows ok and click on the **Save Repository** button to finish adding chart repo.\n\n![Save Repository](https://cdn-images-1.medium.com/max/800/0*tS9uArAROjoOLc05.png)\n\n## Adding OpenEBS to Your Kubernetes Cluster\n\nFirst, make sure your cluster and all nodes are up.\n\nOn the **Control Plane** tab click on your cluster name **OpenEBS Demo**.\n\n![Control Plane tab](https://cdn-images-1.medium.com/max/800/0*0wxTlbbO_yPMJZ8F.png)\n\nOnce the Kubernetes cluster is up on AWS with functional Helm, click on the **Solutions** tab and **Add Solution** button.\n\n![Solutions tab](https://cdn-images-1.medium.com/max/800/0*QofakUAHAb_DRYWp.png)\n\nAdd the solution with the following details:\n\n– **namespace :** default \n– **values -> rbacEnabled :** false \n\n![Install OpenEBS into your cluster](https://cdn-images-1.medium.com/max/800/0*JiSAsRHf5SND0Cbp.png)\n\nClick on **Install** to finally add OpenEBS into your cluster.\n\nState field should be green after OpenEBS is successfully added.\n\n![OpenEBS post install screenshot](https://cdn-images-1.medium.com/max/800/0*1nY357dtw3PNOfAi.png)\n\nNow your cluster is ready; you can run your workloads on openebs-standard storage class.\n\nTo confirm, click on **K8s Dashboard**. This will bring up your Kubernetes Dashboard UI in a new window. You should be able to find the **openebs-standard** option under **Storage Classes**.\n\n![ K8s Dashboard](https://cdn-images-1.medium.com/max/800/0*E5eYS81HcguHaG1r.png)\n\nI’ll cover some workload examples such as MongoDB, Percona, Cassandra, and [Postgres](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/) running OpenEBS on my next blogs (stay tuned).\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-install-openebs-on-aws-using-stackpointcloud/)*.\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-aws-using-stackpointcloud?"},{"id":116,"title":"How to Install OpenEBS with Kubernetes using MiniKube","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"22-10-2017","tags":["Container","Docker","minikube","Kubernetes","Solutions","OpenEBS"],"excerpt":"Whether you are a newbie to Kubernetes looking for a small setup to start or a developer who uses Kubernetes on a daily basis, minikube is the tool that helps you quickly set up and run a Kubernetes environment locally. ","content":"\n## What is MiniKube?\n\nWhether you are a newbie to Kubernetes looking for a small setup to start or a developer who uses Kubernetes on a daily basis, minikube is the tool that helps you quickly set up and run a Kubernetes environment locally. minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.\n\nThere are several options available for developers to install minikube based on an operating system. You can read the detailed instructions for the three most popular operating systems in [minikube Setup](https://github.com/kubernetes/minikube).\n\nHowever, if you are already an experienced minikube user, skip the minikube setup instructions and jump directly to the **Setup OpenEBS** section.\n\nIn this post, I will explain how to set up Kubernetes using minikube directly on Ubuntu 16.04 (without using any VM drivers) and how to configure OpenEBS in hyper-converged mode or, more accurately, create your Container-Converged Infrastructure using OpenEBS Container Attached Storage (CAS).\n\n## Prerequisites\n\nMinimum requirements for minikube:\n\n### Hardware\n\n- Machine Type — minimum 4 vCPUs.\n- RAM — minimum 4 GB.\n- VT-x/AMD-v virtualization must be enabled in your system BIOS\n\n### Software\n\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- If using macOS:\n- xhyve driver, [VirtualBox](https://www.virtualbox.org/wiki/Downloads), or VMware Fusion.\n- If using Linux:\n- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or KVM.\n\n**NOTE:** minikube supports the `-vm-driver=none` option that runs Kubernetes components on the host and not in a VM. Docker is required to use this driver, but no the hypervisor.\n\n- If using Windows:\n- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or Hyper-V. VMware Workstation is not supported.\n\nSince VirtualBox is available on all three platforms, I will describe this option.\n\n## Install VirtualBox\n\nI will not cover the details of VirtualBox installation since it is very common and instructions are widely available online.\n\n1. Go to the [Virtualbox website](https://www.virtualbox.org/wiki/Downloads).\n2. Download and install the binaries required for your operating system.\n\nMake sure that you install [VirtualBox 5.2.0 Oracle VM VirtualBox Extension Pack](http://download.virtualbox.org/virtualbox/5.2.0/Oracle_VM_VirtualBox_Extension_Pack-5.2.0-118431.vbox-extpack) as well.\n\nWhen I was writing this blog post, the most current version was VirtualBox-5.2.0–118431.\n\nOnce VirtualBox is installed, you will see a screen similar to the following:\n\n![Post VirtualBox install screenshot](https://cdn-images-1.medium.com/max/800/0*HztM26xqSWKiYaIx.png)\n\n**NOTE:** You can also use KVM, Hyper-V, and VMware Fusion.\n\n## Install Ubuntu\n\nCreate a new VM with 4 vCPUs, 4Gb memory, and 10GB disk space.\n\n![Creating VM](https://cdn-images-1.medium.com/max/800/0*8wqBzAyAPf_LsbFk.png)\n\nDownload your preferred version of [Ubuntu](https://www.ubuntu.com/download). I will be using Ubuntu 16.04.3 LTS.\n\nUnder **VM Settings/Storage**, mount your ISO image and power on the VM.\n\nInstall Ubuntu with default options. I used *openebs/password* as username/password for simplicity. If you use something else make sure to replace it with yours when you follow the instructions.\n\nFinally login to your Ubuntu VM.\n\nOn your Ubuntu host, install the SSH server:\n\n sudo apt-get install openssh-server\n\nNow you should be able to access your VM using SSH. Check the status by running:\n\n sudo service ssh status\n\n![Accessing your VM using SSH](https://cdn-images-1.medium.com/max/800/0*1rUwIrG2T0EzoBJj.png)\n\nDisable firewall on your Ubuntu VM by running:\n\n sudo ufw disable\n \n\nInstall curl if it’s not already installed:\n\n sudo apt install curl\n\nBy default, for each virtual machine, VirtualBox creates a private network (10.0.2.x) which is connected to your laptop’s network using NAT. However, you may not be able to your VMs from your localhost through SSH just yet. To access your VM, you need to configure port forwarding. In the network setting of the VM. Click on **Advanced/Port Forwarding** and create a rule with the **Host port 3022 **and **Guest Port 22**. Name it *SSH* and leave other fields blank.\n\n![Configuring port forwarding](https://cdn-images-1.medium.com/max/800/0*uDKLTcZapcEZfK3E.png)\n\nNow you can connect to your Ubuntu VM from your laptop using SSH with localhost as the address and port 3022 instead of 22. Connect to your Ubuntu VM using the following credentials: `openebs/password`\n\n## Install Docker\n\nTo get the latest version of Docker, install it from the official Docker repository.\n\nOn your Ubuntu VM, run the following commands:\n\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable”\n sudo apt-get update\n\n![Install the latest version of Docker](https://cdn-images-1.medium.com/max/800/0*-4QRgWvjit9qyKaq.png)\n\nConfirm that you want to install the binaries from the Docker repository instead of the default Ubuntu repository by running:\n\n sudo apt-get install -y docker-ce\n\n![Install the binaries from the Docker repository](https://cdn-images-1.medium.com/max/800/0*Hh8nFvl7xArgJnN-.png)\n\nInstall Docker and make sure it’s up and running after installation is complete:\n\n sudo apt-get install -y docker-ce\n sudo systemctl status docker\n\n![Install Docker](https://cdn-images-1.medium.com/max/800/0*NTvaIXL4LiPakwEy.png)\n\n## Add iSCSI Support\n\nOpenEBS uses iSCSI to connect to the block volumes. Therefore, you need to install the `open-iscsi` package on your Ubuntu machine.\n\nOn your Ubuntu host, run:\n\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n\n![Install the open-iscsi package on your Ubuntu machine](https://cdn-images-1.medium.com/max/800/0*OmIy-bxY3PrD_HYT.png)\n\nCheck that the iSCSI initiator name is configured:\n\n sudo cat /etc/iscsi/initiatorname.iscsi\n\nVerify the iSCSI service is up and running:\n\n sudo service open-iscsi status\n\n![Check that the iSCSI initiator name is configured](https://cdn-images-1.medium.com/max/800/0*30EupY6kOMa30SMj.png)\n\n## Set up minikube and kubectl\n\nOn your Ubuntu host, install minikube by running:\n\n curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\n chmod +x minikube\n sudo mv minikube /usr/local/bin/\n\n![On Ubuntu host, install minikub](https://cdn-images-1.medium.com/max/800/0*62DCuwG4tX8iU_AX.png)\n\nInstall kubectl:\n\n curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl\n chmod +x kubectl\n sudo mv kubectl /usr/local/bin/\n\n![Install kubectl](https://cdn-images-1.medium.com/max/800/0*9jZx-rusvrdn9mEe.png)\n\nSet up directories for storing minkube and kubectl configurations:\n\n mkdir $HOME/.kube || true touch $HOME/.kube/config\n\nSet up an environment for minikube by adding the following lines to the end of the `~/.profile` file:\n\n export MINIKUBE_WANTUPDATENOTIFICATION=false\n export MINIKUBE_WANTREPORTERRORPROMPT=false\n export MINIKUBE_HOME=$HOME\n export CHANGE_MINIKUBE_NONE_USER=true\n export KUBECONFIG=$HOME/.kube/config\n\nConfirm that environment variables are saved in your profile file:\n\n cat ~/.profile\n\n![Confirm that environment variables are saved in your profile file](https://cdn-images-1.medium.com/max/800/0*rxjoxM6qkYkppd5h.png)\n\nStart minikube:\n\n sudo -E minikube start — vm-driver=none\n\n![Start minikube](https://cdn-images-1.medium.com/max/800/0*UaQ_6Y2m4hv6P4oc.png)\n\nIf you forgot to install Docker, you will get the following error:\n\n![Error screenshot when forgot to install Docker](https://cdn-images-1.medium.com/max/800/0*ysp8RnG5DWDu_Q0j.png)\n\nWhen using the none driver, the kubectl config and credentials generated will be root-owned and will appear in the root home directory. To fix this, set the correct permissions:\n\n sudo chown -R $USER $HOME/.kube\n sudo chgrp -R $USER $HOME/.kube\n sudo chown -R $USER $HOME/.minikube\n sudo chgrp -R $USER $HOME/.minikube\n\n## Verify minikube configuration\n\nVerify that minikube is configured correctly and it has started by running:\n\n minikube status\n\n**Example:**\n\n![Verify minikube configuration](https://cdn-images-1.medium.com/max/800/0*yK3Wlyy81I15tNZp.png)\n\n**Note**\n\n- If the minikube status displays **Stopped**, add the `sudo minikube start` command.\n- If you forgot to set the permissions, minikube will display errors indicating permissions denied to configuration files, fix the permissions by running the following commands:\n\n sudo chown -R $USER $HOME/.kube\n sudo chgrp -R $USER $HOME/.kube\n sudo chown -R $USER $HOME/.minikube\n sudo chgrp -R $USER $HOME/.minikube\n\n## Verify Kubernetes configuration\n\nCheck that kubectl is configured and services are up and running by getting the list of Kubernetes nodes and pods:\n\n kubectl get nodes\n kubectl get pods — all-namespaces\n\n![Verify Kubernetes configuration](https://cdn-images-1.medium.com/max/800/0*noWgoiv0GLk43BRB.png)\n\n## Set up OpenEBS\n\nDownload the latest OpenEBS Operator files using the following commands:\n\n git clone https://github.com/openebs/openebs.git\n cd openebs/k8s\n\n![Download the latest OpenEBS Operator files](https://cdn-images-1.medium.com/max/800/0*UNKK2cZhYPJVbTDx.png)\n\nBy default, OpenEBS launches OpenEBS Volumes with two replicas. To set one replica, as is the case with a single-node Kubernetes cluster, in the openebs-operator.yaml file, specify the environment variable `DEFAULT_REPLICA_COUNT=1`. This is supported in OpenEBS version 0.4 onward.\n\n![Replica count screenshot](https://cdn-images-1.medium.com/max/800/0*SxXEzbDmpVA5ZhwS.png)\n\nApply the configuration changes:\n\n kubectl apply -f openebs-operator.yaml\n\n![Applying configuration change](https://cdn-images-1.medium.com/max/800/0*WB16UScHye4LClft.png)\n\nAdd the OpenEBS storage classes that can then be used by developers and applications:\n\n kubectl apply -f openebs-storageclasses.yaml\n\n![Adding the OpenEBS storage classes](https://cdn-images-1.medium.com/max/800/0*mojNYfZbll-g6bdk.png)\n\n#### Running stateful applications with OpenEBS storage\n\nTo use OpenEBS as persistent storage for your stateful workloads, set the storage class in the Persistent Volume Claim (PVC) of your application to one of the OpenEBS storage class.\n\nGet the list of storage classes using the following command. Choose the storage class that best suits your application.\n\n kubectl get sc\n\n![Getting the list of storage classes](https://cdn-images-1.medium.com/max/800/0*artfNnT8fZSziaKH.png)\n\nYou can find samples of YAML files for stateful workloads using OpenEBS under the `openebs/k8s/demo` folder.\n\n![Finding samples of YAML files for stateful workloads](https://cdn-images-1.medium.com/max/800/0*KCn5Z4-av-7Hevj_.png)\n\nNow you have your Kubernetes cluster up and running. In my next blog posts, I will cover the installation of stateful workloads such as Cassandra and [PostgreSQL](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/), as well as the benefits of running your stateful workloads on OpenEBS. Stay tuned!\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-install-openebs-with-kubernetes-using-minikube/)*.\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-with-kubernetes-using-minikube"},{"id":117,"title":"Why are users using OpenEBS *on* EBS?","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"19-10-2017","tags":["Container Native Storage","Featured","MySQL","Kubernetes","OpenEBS"],"excerpt":"We were a little surprised to see OpenEBS started to be used on top of EBS itself. So we dug in and asked why?","content":"\nWe were a little surprised to see OpenEBS started to be used on top of EBS itself. So we dug in and asked why?\n\nThe following lays out what we learned, focusing mostly on the most common use case.\n\nAs you likely know, broadly speaking, there are two types of stateful applications. We call them Mercy Apps and NoMercy apps (Not generic names, but we started using these names in the OpenEBS community discussions 🙂)\n\n1. **Mercy Apps** — The apps deal with data resiliency at the application level; for example, they synchronously replicate and disperse the data and will have less dependency on the high availability of storage underneath. Some examples of mercy apps are Cassandra and even (usually) MongoDB.\n2. **NoMercy Apps** — The apps that have a single copy of the data and have no idea of synchronous replication of data. These applications assume that the underlying storage is always (at least highly) available. The most famous example for NoMercy apps is the famous “MySQL server”.\n\nToday, most of the legacy apps are NoMercy Apps. Also, MySQL or Postgres tends to be the first choice for most of the developers and remains the most commonly used database for quick application development and deployment. The applications that use MySQL underneath are being containerized and probably being moved to the cloud container services such as Amazon ECS. It is important to note that even NoSql databases like MongoDB that protect the data across nodes are often deployed as a single copy (as a NoMercy App) and that we are seeing cases where users of NoSql use the storage for node pre-population for example instead of relying on node rebalancing at the application / NoSql level.\n\n![Shifting from legacy to microservices](https://cdn-images-1.medium.com/max/800/1*Bayd4nQST787TIbYo_5aWg.png) \n(***Shifting from legacy to microservices***)\n\nOnce you lift and shift your legacy app onto containers and the cloud, you ideally want to allow for Kubernetes to orchestrate the dynamism of container movements among various hosts; while this dynamism is core to the value of containerization, it also creates a unique challenge for the availability of data/storage to the application.\n\n![EBS volume may not be immediately available](https://cdn-images-1.medium.com/max/800/1*ISz4kvGREGlXZkBwiwSRjQ.png) \n(***EBS volume may not be immediately available***) \n\nAs the above drawing suggests, we learned from some OpenEBS community users that when app containers move from one host to another, the time it takes to detach the EBS volume from one host and to attach the same EBS volume to the new host can *cause downtime for the stateful application*.\n\n### There are two solutions to this problem.\n\n***First solution:*** Re-architect your application and make it a mercy application. Use the new age databases like Cassandra or MongoDB and configure them to protect the data at the application layer, which of course needs special training and quite a bit of work. Even the flavor of SQL support may change, so you may find yourself rewriting your queries, always a source of fun and enjoyment 🙂\n\n![Sync replicate at the DB level](https://cdn-images-1.medium.com/max/800/1*rdabUhTkx6iF3Ncv3EKlrQ.png) \n(***Sync replicate at the DB level***)\n\nBtw, if the thought of moving to a new Database is daunting, you would be interested in using plug-in code to the existing no-mercy apps to perform the synchronous replication. In the case of MySQL, one option is to migrate to MariaDB and then use the Galera sync replication plugin. You would be lucky if you already have a plugin that is resilient and doesn’t add performance overheads.\n\n***A second solution*** — that is probably the top reason users are running OpenEBS and similar containerized storage controllers ON EBS — is a more elegant one : *protect the data at the storage layer*. By putting OpenEBS into your pods as the provider of storage and letting it handle data placement on the local nodes (in this case one or more, yes, EBS volumes) while also replicating the data per the policies you prefer, you avoid rewiring your application.\n\n![Using OpenEBS for high availability of Mysql DB data on AWS EBS](https://cdn-images-1.medium.com/max/800/1*3npgXXxGEOFD4uh_KRvPng.png) \n(***Using OpenEBS for high availability of Mysql DB data on AWS EBS***)\n\nOk, great, however — you are now doing storage differently. Isn’t that itself a challenge?\n\nWell, if you are already moving towards Kubernetes then you are already learning the skills needed to run OpenEBS. OpenEBS is integrated into the Kubernetes storage architecture to make the volume provisioning basically the same behind the scenes experience as attaching the EBS volume for a given pod. Instead of attaching the AWS EBS volume to the application pod using kubernetes.io/aws-ebs provisioner, the developer simply uses kubernetes.io/openebs provisioner. The underlying pieces such as integrating the AWS EBS volume into OpenEBS volume are handled by the OpenEBS provisioner.\n\nMileage may of course vary. We are building OpenEBS in the open in part to get feedback from users like those that prompted me to write this blog in the first place. Please share with the community experience of running OpenEBS on your Kubernetes based AWS ECS — whether your applications are Mercy or NoMercy. You can find the instructions and help [here](http://openebs.readthedocs.io/en/latest/install/cloud_solutions.html#amazon-cloud). And — yes — we welcome other bloggers 🙂 We’ve seen a huge ramp in issues and PRs and so forth in the last couple of months. The next step might be user blogging — please feel free and we’ll help if useful.\n","notHasFeatureImage":true,"slug":"why-are-users-using-openebs-*on*-ebs?"},{"id":118,"title":"How to build an e2e?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"17-10-2017","tags":["Cloud Storage","E2e Testing","Featured","Kubernetes","Storage"],"excerpt":"e2e which expands into end to end speaks for itself. It can be treated as component testing, integration testing, or something that tries to test stuff outside the scope of unit tests.","content":"\n### What is e2e?\n\nWell *e2e* which expands into end to end speaks for itself. It can be treated as component testing, integration testing, or something that tries to test stuff outside the scope of unit tests. I got attracted to this side of development when I was surprised looking at one of the smallest possible directory names ever in a code-based project. As I speak, we get to see a number of Go based projects having an *e2e* folder. I believe, it has been popularized by the likes of etcd, Kubernetes, and perhaps a few other open source communities.\n\n### How to build one?\n\nBefore even getting into the How’s part, we must rather be comfortable with following questions:\n\n— *Do we need to?* \n — *Does our project need one?*\n\nMore often than not, a code based project will love to have one. Perhaps not with this name but the need will definitely exist. Some of the guiding principles highlighted in this article will also help us in getting the answers to these questions. So keep reading.\n\n### Assuming our project needs one, how to build it then?\n\nAre there any guidelines/rules that we can follow? Here they are:\n\n— ***`Refer and re-use wherever possible than build from scratch`*** is the golden rule.\n\n— ***`Start Small`***. In this context, it refers to '*not to gulp the entire code base and be very selective even to the point of refusing 90% of the code that is available for free*'. This is the learning I have had all these years as a programmer. This is essential if one wants to reach the milestones in time which will otherwise become a mirage. I have burnt my fingers multiple times with regards to this principle. The latest one was when I tried to reuse the entire Kubernetes e2e for [Maya’s](https://github.com/openebs/maya/) e2e. Needless to say I failed miserably. Some of these reasons are explained in this article.\n\n### Is that all?\n\nDefinitely not. We will soon get into the How’s part which in turn is a repository of queries, doubts, and concerns.\n\nTo clarify it further, I had these queries when I started with the e2e journey for the [Maya](https://github.com/openebs/maya/) project.\n\n- Should e2e be shell wrappers over [CLI](https://en.wikipedia.org/wiki/Command-line_interface), for example, [kubectl](https://kubernetes.io/docs/user-guide/kubectl-overview/) & [mayactl](https://github.com/openebs/maya/tree/master/cmd/mayactl)?\n- Should it be tied to Go [*testing*](https://golang.org/pkg/testing/) library? Any benefits?\n- Should it use the [*Ginkgo*](https://onsi.github.io/ginkgo/) library? Anything to gain?\n- Should it be moulded with [*Ansible*](https://www.ansible.com/) and let the scene get enacted by its players (*read playbooks*)?\n\n### Simplicity — One Rule to Rule them All\n\nAll these queries should be answered with simplicity in mind. In other words, how to construct test code that is simple?\n\nWith simplicity in mind, let us pen down some dos and don’ts that will be an\nindicator of simplicity versus complexity.\n\n### Simplicity — Direct Thinking!!\n\nListed are some of the direct modes of thinking with regards to simplicity.\n\n- Code should just try to eliminate the repeating tasks of the developer, tester, or operator. It will be simpler if it builds logic on only those ingredients that play an inhibiting role in manual testing.\n- Find out one tool (*in other words a dependency*) that can make this test code easy to reason and comprehend. Build the code around this dependency. Though this dependency becomes a hard requirement, we are still good if it satisfies the simplicity rule.\n- It should be limited in its scope. For example, in the case of [Maya](https://github.com/openebs/maya/), its e2e is scoped to Kubernetes. Maya e2e avoids [*Ginkgo*](https://onsi.github.io/ginkgo/) etc libraries. It also avoids *Ansible* as the latter is not at all required.\n- It should be built using a high level language. Maya e2e uses Go as its only programming language. It does not use shell or any other scripts. This choice is also dependent on the scope the test code is targeting at. In the case of Maya, [Kubernetes](https://kubernetes.io/) (*and host of other container orchestrators*) use Go as their primary language. This helps Maya to abide by the golden rules mentioned earlier.\n\n### Simplicity — Inverted Thinking!!\n\nListed are some of the inverted modes of thinking with regards to simplicity.\n\n- It should not try to be another [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) in the making. In addition, it should not deal with some smart syntax. It will instantly seem smart to the eyes which developed it, but will definitely repel others.\n- It should not get into the way of the developer or the tester in form of auto-generation of test code etc. This will lead to its brittleness.\n- It should not dedicate its logic to concurrency while running the test cases. It should not build its logic around CPU cores and parallelism for running the test cases either. It might seem concurrency is a required feature for a particular test case. However, do we take concurrency into factor while executing these cases manually? The test logic can have the concurrency built later in an iterative fashion. Remember “*shipping is better than perfect”* in these cases.\n- It should not play around with the abstraction of containers and their orchestrators. Remember this is not the reason for its existence in the first place. It goes against the prescription of simplicity. Hence, cut this noise if they are not a natural fit.\n- It should not mandate running the tests in containers. It is considered cool these days to run test cases in containers. However, it can backfire. For example, running your app and test cases as containers within a namespace or inside the same cluster. We do not want *100s* and *1000s* of containers getting spawned in the same setup that is meant to test your app which is again a set of containers.\n- It should not involve a learning curve. It should be an involving library that can take inputs from the team (*internal as well as external*) and evolve.\n- It `need not / must not` follow design patterns. The best it can do is to adhere to the core language’s best practices.\n\n### What else should the e2e try to achieve?\n\nAnswers to these questions will help you in getting the other aspects of your e2e.\n\n- Did we avoid fancy scripting ?\n- Is e2e (*especially the reusable code pieces*) better than the one it was created from?\n- Will *`anybody be able to contribute to anything`* in this e2e project?\n- Does it look similar to the practices followed by its programming language? Why? This leads to a 0 learning curve and hence is simpler to understand?\n- Do you still remember the `Golden Rules`? Add another one & that is **`End Small`**.\n\n*Thanks to Madhuri Hebbar and Uday Kiran.*\n","notHasFeatureImage":true,"slug":"how-to-build-an-e2e?"},{"id":119,"title":"Quick update on OpenEBS v0.4 — a developer friendly release!!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"03-10-2017","tags":["Container Native Storage","Container Orchestration","minikube","Open Source","Updates","OpenEBS"],"excerpt":"OpenEBS v0.4.0 is out and I take pride in sharing that it is powered by the same set of tools, used by communities that have adopted DevOps DNA for delivering products.","content":"\n[OpenEBS v0.4.0](https://github.com/openebs/openebs/releases/tag/v0.4.0) is out and I take pride in sharing that it is powered by the same set of tools, used by communities that have adopted DevOps DNA for delivering products.\n\nPersonally, for me, the most exciting part of this release is that it involves contributions from community members across the globe. We have crossed more than 100 Pull Requests from 25+ contributors.\n\nOur decision to remain OpenSource is paying off!! As new contributors come on board, the following DevOps tools are helping us to sustain the quality:\n\n- [Github ](https://github.com/openebs/openebs/issues)for collaboration — managing code reviews, releases, and now project management (issues and milestones).\n- [Travis ](https://travis-ci.org/openebs/) and [DockerHub ](https://hub.docker.com/r/openebs/)— validate the code commits and release new docker images\n- [Jenkins and Ansible](https://github.com/openebs/openebs/tree/master/e2e) — run the e2e tests on the new docker images.\n- Static Analysis is provided through a wide range of tools like gofmt, flake8, codecov, goreport, and the list needs to expand.\n- [ReadTheDocs ](http://openebs.readthedocs.io/en/latest/index.html) — updating the live documentation site\n\nOpenEBS, like other storage options (EBS, Rook, and others), is already deeply integrated with Kubernetes and is now part of the [kubernetes-incubator project](https://github.com/kubernetes-incubator/external-storage/tree/master/openebs). OpenEBS also provides similar intuitive mechanisms to provide block storage to your stateful application on Kubernetes using concepts like:\n\n- StorageClasses\n- PersistentVolumeClaims\n- PersistentVolumes\n- DynamicProvisioner\n\nIn addition, unlike others, OpenEBS delivers container-native storage by *using Kubernetes (as opposed to running on Kubernetes)* itself as the underlying framework for scheduling and storing configuration data. There are also efforts underway for the upcoming release to make use of Kubernetes *LocalStorageManager, Kube-Dashboard, and CNCF projects like Prometheus, FluentD, Grafana, Jaegar*, etc., for managing and monitoring the storage functionality.\n\nYou can easily get OpenEBS running on your Kubernetes Cluster with the following two commands and then point your application’s PVC to one of the OpenEBS Storage Classes.\n\n`kubectl apply -f openebs-operator.yaml` \n`kubectl apply -f openebs-storageclasses.yaml`\n\nThe above YAML files can be downloaded from [here](https://github.com/openebs/openebs/tree/master/k8s). For detailed instructions refer to our [quick start guide](http://openebs.readthedocs.io/en/latest/getting_started/quick_install.html) or checkout our [sample stateful applications](http://openebs.readthedocs.io/en/latest/install/install_usecases.html) which include Percona, Jupyter, Postgresql, etc.,\n\nWhile you are at the documentation, you can also check out the additional deployment options that we have added with this release:\n\n- [Running Kubernetes and OpenEBS on AWS](http://openebs.readthedocs.io/en/latest/install/cloud_solutions.html)\n- [Running Kubernetes within minikube](http://openebs.readthedocs.io/en/latest/install/dev_solutions.html)\n\nThough we are still in the first leg of our journey, with OpenEBS v0.4, you get usable container-native storage, with enterprise storage capabilities like detailed *Volume IO statistics* and *Snapshots*.\n\n*Please refer to the [CHANGELOG](http://openebs.readthedocs.io/en/latest/release_notes/releasenotes.html), for a summary of updates in v0.4 and [ISSUELOG](https://github.com/issues?q=user%3Aopenebs+and+is%3Apr+and+merged%3A%3E2017-06-23+sort%3Acreated-asc)for the list of Pull Requests.*\n\nAnd we are always looking for help from OpenSource savvy community members. You can contribute in several ways — take your pick from our growing [task list](https://github.com/openebs/openebs/labels). Join us on [#Slack](http://slack.openebs.io/).\n","notHasFeatureImage":false,"slug":"quick-update-on-openebs-v0.4-—-a-developer-friendly-release!!"},{"id":120,"title":"Not Yet Another Distributed Storage System","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"10-09-2017","tags":["Kubernetes","Docker","Container"],"excerpt":"These days, it seems that a lot of storage vendors are taking a scale out approach to delivering high-performance storage to meet the increasing demand for IOPS and bandwidth.","content":"\nThese days, it seems that a lot of storage vendors are taking a scale-out approach to deliver high-performance storage to meet the increasing demand for IOPS and bandwidth. Reduced latency is also high on the storage requirement list, however, scale-out distributed systems typically result in the inverse, i.e., it increases latency.\n\nAlthough complex distributed systems are easier to build these days due to a variety of factors including maturing software implementations of the likes of Paxos, distributed hash tables, and RAFT it seems that creating a storage system that utilizes these concepts and is easy to manage and maintain in production — is not.\n\nOn the other hand, if you want to leverage the distributed nature of storage at the client level, more often than not, the end-users are required to run specialized clients/drivers to unleash the enormous bandwidth these systems can deliver — and so now your storage has infected your client which makes it even less attractive.\n\nArguably, if you need hundreds of gigabytes of throughput for a particular workload, the Linux kernel comes preloaded with [one](http://www.orangefs.org/). So you have to look no further from a tech support side of things — as what better experts to find that concern themselves with the Linux kernel and thus OrangeFS, right? No need to search any further, right?\n\nLet’s first try to understand why a lot of people I’ve spoken to in the past typically chose **“scale-out.”** It seems they all like the idea of “**add another box”** to add performance and capacity. The fact that you can’t scale performance decoupled from capacity is what they take for granted. However, due to economics forcing IT segments to do things cheaper and more predictably, this is not the case — anymore.\n\nSome storage vendors have found a solution for this, by not selling you expensive boxes but rather just the software. You simply scale in any cloud on any hardware or so they say. However, this **“any any”** approach does not fit in with the requirement to make things more predictable in fact quite the opposite.\n\nSo if we summarize the downsides:\n\n- Distributed storage difficult to develop and is hard and nasty to manage in production\n- Specialized drivers needed to unleash the real potential\n- Scaling the number of nodes does not decrease latency, in fact, usually the opposite (depending on implementation)\n- Best scales bandwidth and IOPS, however, this is not what a typical workload requires\n- Complex consistency models create surprises regarding what is on disk or not\n- Not especially good at leveraging flash or NVMe to deliver great performance and a small footprint\n- Big blast radius — the more data you can put in one system, the more you might lose or at least lose access to when you most need it\n- The complexity and need for quick metadata updates across nodes both argue against multi-cloud deployments (though there are some scale-out file systems being built that claim to address these issues)\n\nNow there will always be vendors out there that claim to have solved it all, unlimited scale-out, never ending IOPS, more bandwidth than the whole of the internet combined, and latency — sure we do that too…\n\nAt [OpenEBS](https://www.openebs.io/), however, we took a different approach by not trying to solve the distributed problems but to take a step back and try to determine what are the real problems people need to solve?\n\nSpeaking to our early tech-preview customers, we were shocked with awe to see that due to the complexity of storage these days, they simply revert to [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage) (DAS). “It won’t go any quicker than that”, they say. And you have to make a very, very, strong case to argue against that. In fact, come to think of it, it is nearly impossible if you consider the speed of NMVe devices.\n\nHas the storage market become so consumed with itself that it keeps making storage products so complex, with each vendor, having its own **“if — and — or buts”** that they are fed up with it and revert back to DAS? Did it become so unpredictable? The time of consolidated storage churn?\n\nYes, the storage market is consumed with itself. But there is more, the storage **needs** have also changed. In the early SAN days, it was about consolidating islands of storage into a bigger one as it would be easier to manage and the performance of the SAN would be higher than some individual devices. Virtual machines made it possible to consolidate compute which also made a lot of sense as boxes were mostly idle.\n\nHowever, all of this work was done for one reason and one only — to accommodate the piece of software that matters the most: the app.\n\nThe app is central and is the only thing that matters; everything around it is inflicted upon us.\n\nNobody wakes up one day and says, “I need to get myself a SAN” — it was likely the best of the worst options at that time.\n\nAs DevOps happened alongside containerization, the application itself has become a distributed system. A distributed system in the sense that subcomponents of the app as a whole may run on the same box, different box, and are loosely coupled by APIs one way or the other and working together to solve a complex (business) problem.\n\nAs applications have become distributed systems themselves, they have become easier to scale and thus don’t always require high IOPS low latency storage devices to scale performance-wise. So storage and capacity are now loosely coupled and suffer from data gravity in a different way than they once did. Additionally, data availability does not solely depend anymore on expensive storage arrays as applications are designed to replicate their data straight out of the GIT. As these apps are distributed by nature, we think that a distributed storage system is not only complex and nasty but completely unneeded.\n\nSo — what’s a storage vendor to do? Well, we have it easier, as we don’t have much legacy — though we do have lots of experience building storage on containers as a part of our [ElastiStor](http://www.cloudbyte.com/products/elastistor-os/) product. And so we were able to start with the customer questions discussed above — in the age of microservices and cloud and containers, what job are they looking to do regarding serving, moving, and protecting their data? And we can answer those questions in a way that is entirely free from whether it is storage strategy A, B, or C. In future blogs we’ll talk about OpenEBS more (of course) — and maybe more importantly we would like to discuss with you how we think the job storage is being asked to do has changed and hence, how and why the old storage industry may be coming to an end.\n\nPlease, feel free to join us on [slack](http://slack.openebs.io/) to discuss in a more real-time fashion.\n","notHasFeatureImage":true,"slug":"not-yet-another-distributed-storage-system"},{"id":121,"title":"ARA - Recording Ansible Playbook Runs","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"31-08-2017","tags":["Ansible","Automation","DevOps","OpenEBS"],"excerpt":"Ansible playbooks can generate quite a lot of console data. Add the -v (I do it by default!) and you have quite a lot to scroll through.","content":"\nAnsible playbooks can generate quite a lot of console data. Add the -v (I do it by default!) and you have quite a lot to scroll through. Oftentimes, one tends to feel the need for a better reporting mechanism — one which is easy to scour for specific task status, whilst having the luxury to extract additional debug info for the same if needed. Also, it would be great if this aid is available for playbook runs across time, i.e., for older playbook runs as well. This would be especially beneficial when running ansible-based CI suites, like @OpenEBS, where the application deployments, test setup & execution are driven by Ansible.\n\nSounds like a perfect requirement for a UI-based solution, doesn’t it?\n\nA lot of people using Ansible for medium-large scale deployments are known to use **Tower**, **Rundeck**, or the opensource alternative **semaphore** to achieve this (Tower has a lifelong self-support trial license without a few features for under 10 nodes). There is also **Foreman**, which is great in environments where a hybrid tool set — a puppet with ansible, or chef with ansible, etc., is used. Most of these are workflow management tools that do more than *just* what we desired above, i.e, playbook recording. These tools provide a centralized management capability wherein inventory, users, task/play scheduling, notifications can all be controlled from a dashboard. Now, it is also possible to integrate these with other popular CI tools like Jenkins (watch [this](https://www.youtube.com/watch?v=CqjeIiHvy30&feature=youtu.be)).\n\nHowever, if there is already a system in place to perform some of the things these tools do (dynamic inventories, playbook triggers/scheduling) while not really needing others (access control, graphs), and you are only looking for the ability to store and analyze playbook runs via UI, then **ARA** (Ansible Run Analysis) is your tool.\n\n![ARA (Ansible Run Analysis)](/images/blog/ansible-run-analysis.png)\n\nHeavily used by the OpenStack community in their CI projects, ARA is built to just *\"record\"* playbook runs (Read ARA’s [manifesto](http://ara.readthedocs.io/en/latest/manifesto.html#manifesto), to understand more about its narrow focus). It does this via an ansible callback plugin to store run details into a database and a web interface to visualize the database.\n\n![Viewing playbook tasks summary in ARA](/images/blog/playbook-tasks-summary-in-ara.png) \n(***Viewing playbook tasks summary in ARA!***)\n\n![Viewing task details in ARA](/images/blog/viewing-tasks-details-in-ara.png) \n(***Viewing task details in ARA***)\n\nWhile it uses SQLite and an embedded webserver, respectively, for these purposes, you could even customize it to use MySQL & Apache. The UI includes nifty features like host fact lookup, playbook params page, search filter, property based sort, and link to code snippets!\n\n![ARA code snippet](/images/blog/see-specific-task-ran.png) \n(***Click on action to see where the specific task ran***)\n\nA nice video explaining the web interface, with playbooks from the OpenStack-Ansible project is [here](https://www.youtube.com/watch?v=k3i8VPCanGo)\n\nIn addition to these, ARA also provides\n\n- Couple of Ansible modules for persisting & viewing some user data that one may want to view on the browser-based UI (like an ansible “fact”, but for visualization purposes :P).\n- A CLI to query the database (While I haven’t found much use for it till now, it is useful to custom create some reports)\n\nARA follows the same support cycle as the upstream Ansible community and is under active development (See [github](https://github.com/openstack/ara))\n\nHere is an [ansible role](https://github.com/openebs/openebs/tree/master/e2e/ansible/roles/ara) we have written that you could use to quickly setup ARA on your ubuntu box 🙂\n\nWe @OpenEBS, have found great benefit in using this tool and would happily recommend it for the use cases discussed. !!\n","notHasFeatureImage":true,"slug":"ara---recording-ansible-playbook-runs"},{"id":122,"title":"Ansible @ OpenEBS — The whys and hows","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"31-07-2017","tags":["Ansible","Ci","DevOps","Jenkins","Vagrant"],"excerpt":"We are using Ansible as one of the critical moving parts of our automated test suite in the CI pipeline @OpenEBS. The question was expected in some ways,","content":"\nDuring a telephone conversation with a former colleague and good friend I was confronted with an interesting question, “I know Ansible is a great configuration management tool, why bend it as a test automation framework?”\n\nWe are using Ansible as one of the critical moving parts of our automated test suite in the CI pipeline @OpenEBS. The question was expected in some ways, what with the friend having spent a few good years working with more “traditional”, proprietary, and hand-built-from-scratch test-automation frameworks based on Perl. The subsequent discussion (mostly answers and follow-up questions) helped me internalize why we chose Ansible at OpenEBS and how better to use it.\n\nFelt this warranted a blog post to make my thoughts public and hey, of course, gain more feedback!\n\nOK, that let the cat out of the bag\n\n## Infrastructure as Code (IaC)\n\nOne of the biggest IT trends over the last few years has been managing infrastructure through automation. One might argue that puppet started way back in 2005, thereby making this practice far older than most believe, but the way it has taken ops departments of most organizations by storm in the past 5–6 years is nothing less than a revolution. In fact, the paradigm of DevOps is built on managing infrastructure as code. And when we say code, the expectation of most ops personnel around the “language” would be that it doesn’t require deep programming knowledge and have a steep learning curve — which is what DSL (Domain Specific Language, sometimes also referred to as Domain Scripting Language) based frameworks like ansible achieve. It also helps that ansible adopts an imperative programming model (using YAML) that works well because of its alignment with the traditional command-based approach of ops teams.\n\n![Ansible GitHub trends ](https://cdn-images-1.medium.com/max/800/1*7Di79EF1SxNqF0F0KD1E-A.jpeg)\n(***Ansible GitHub trends (Courtesy: [https://www.ansible.com/blog/another-good-year-for-ansible-users](https://www.ansible.com/blog/another-good-year-for-ansible-users))*** )\n\nBut, how does the above address our question?\n\n**Answer**: A major portion of the test duration of infrastructure-based software, such as storage software involves “manipulation” of infrastructure. Setting up bare-metal boxes, virtual machines, or containers, installing packages, executing various commands that control & alter system state, monitoring for specific behavior are key aspects of this process. Consider the need to run the above as batch processes and perform parallel execution on multiple nodes — and the inevitability of a workflow orchestrator dawns upon you. Especially so when you are testing a solution like OpenEBS that is designed to provide storage for DevOps use cases (read more about this [here](/blog/storage-infrastructure-as-code-using-openebs))\n\nIs not an approach (and the tool) soaked in “**devops-ness**” a pre-requisite to test the storage solution specifically designed for DevOps use cases 🙂 ?\n\n## Why Ansible, why not chef, puppet, salt, etc.?\n\n![Ansible](https://cdn-images-1.medium.com/max/800/0*NQOK_gId-YBZMe02.png) \n\nErr.., this seems to have been done to death on the internet. Yet, new posts on this topic seem to spring up everytime I look. So, without discussing the why not, let me touch upon the aspects about ansible that appealed most to us.\n\n**Powerful, yet very simple**: Ansible’s power comes from its simplicity. Under the hood, it is just a DSL for a task runner over a secure shell (ssh) with intuitive modules for achieving most (if not all) system functions. The soft learning curve in ansible is one of its major advantages over its rivals (*Ok, I said I won’t do the why-not, but there is a feeling that puppet, chef, etc., are over-designed for the jobs they do.*) In an open-source project like OpenEBS, a need to build a template for the contributors to write their own test workflows without spending too much time was an important consideration.\n\n**Idempotency**: The ansible playbook (a set of tasks written using the modules mentioned previously) when run twice gives the same end result. This is a great help when it comes to reusing testbeds.\n\n**Speed of execution**: One of the benefits of having an agent-less architecture (apart from a complexity-free install and usage experience), ansible playbooks zip through configuration, and other “system” tasks (mostly, test logic) fairly fast. While there are supposedly issues at scale (1000s of nodes), it works just great for our needs. Want to setup a working Kubernetes cluster with OpenEBS storage on-premise in less than 20 min? — check out the playbooks on our [github repo](https://github.com/openebs/openebs/blob/master/e2e/ansible/openebs-on-premise-deployment-guide.md)\n\n**Rich module library**: Ansible has modules for most things under the sun 😐 Nuff said, go look : [Ansible modules](http://docs.ansible.com/ansible/latest/modules_by_category.html)\n\n(*As an aside, this was one of the other reasons why we started using ansible as a test engine, apart from the infrastructure angle*)\n\n**Plugins**: While ansible is great for configuration management, workflow orchestration, etc., it needs to be able to work well with other tools/frameworks that make up the CI-CD pipeline( Jenkins, Vagrant, etc.,) And all these have ansible plugins (How much we use them is a topic for another day, but the point is ansible does have integration if you choose to utilize it). More important than existing integrations is the ease with which you can extend ansible’s capabilities with custom plugins. Python was a dev + ops favourite, even before DevOps became a fad and ansible is built using it -so, there you go!\n\n**Community**: One of the ansible’s biggest strengths is its community. We discussed extending ansible’s capabilities via custom plugins. Chances are you never have to write one, because the community already has two versions of it (*okay, I may be exaggerating*), but the active community makes it a lot easier to adopt. There is a lot of documentation available as well. All of this means, for most purposes, you don’t have to opt for paid support for issues OR add-ons until you really scale or get complex.\n\n## How are we using Ansible?\n\nAt OpenEBS, we are using Ansible as :\n\na) A means to enable rapid deployments of applications in user environments. Today, you can use our Ansible playbooks to get a Percona MySQL server instance or a PostgreSQL statefulset up and running with OpenEBS storage on-premise from plain vanilla VMs in double quick time, with single command execution. Even as I write this, efforts are underway to create playbooks to perform such deployments on the cloud, right from provisioning VM instances to running test loads to verify setup stability.\n\nb) As a “***test orchestrator***”, i.e., for provisioning testbeds, executing test logic, and notifying users. That doesn’t mean we have stopped writing shell scripts or python scripts in QA (there are still functions which one might have to execute via shell or the ansible “shell” module or python, due to lack of actual ansible modules. Not to mention existing scripts which one wouldn’t bother converting into playbooks).\n\nOur current CI workflow involves a Jenkins master polling for updates to git repos, followed by bringing up VMs on-premise using vagrant, configuring those using ansible, followed by execution of test playbooks and user notification on slack. The CI is still evolving and efforts are on to make it more robust — you could join the OpenEBS-CI [slack channel ](http://slack.openebs.io/) & browse the [github pages](https://github.com/openebs/openebs/tree/master/e2e) if you are an enthusiast/would like to contribute!\n\nIn forthcoming blogs, I would like to discuss more on the “***How***” and share thoughts, challenges, solutions around using ansible both as a preferred application deployment mechanism as well as a “test orchestrator”.\n\nThanks for reading !!\n","notHasFeatureImage":true,"slug":"ansible-@-openebs-—-the-whys-and-hows"},{"id":123,"title":"Are you afraid of Go?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"27-07-2017","tags":["Docker","Go","OpenEBS","Programming Languages","Programming Tips","Tutorials"],"excerpt":"Are you an experienced C programmer or perhaps an adept Java hacker finding Go somewhat awkward to deal with.","content":"\nAre you an experienced **C** programmer or perhaps an adept **Java** hacker finding [**Go**](https://github.com/golang/go/wiki/whygo) somewhat awkward to deal with? There may be different elements of surprise or familiarity based on the programmer’s background e.g a Java programmer may relate the GOPATH settings with the settings associated with JAVA_HOME. On a similar note, the very same programmer might be surprised with Makefiles that are used to script the compilation, build & other stuff. I am sure though that this experience will be different for a seasoned C programmer.\n\nIf the above and perhaps multiple other reasons inhibit you from familiarising yourself with Go then you are not the only one. The best thing is there is a similar number of solutions that will help you play along with the Go pitch.\n\nGoing back in time, this thorny feeling was exactly what we had faced when we started [***OpenEBS***](http://openebs.io) development a year back. However one of the core goals of *OpenEBS* is ease of use; whether it is for the admin or the operator or the developer, tester, and so on. In other words, different personas involved during the Software Life cycle, Deployment & Maintenance should not go through a learning curve and rather experience *OpenEBS* as the simplest form of storage software that cuts through all the storage noise. This meant we did not want Go or any other programming language, as a matter of fact, to limit our way towards this enlightened journey.\n\nDuring those days we implemented some of the strategies that helped nullify the above pain points and provide a smooth cruise to all current & future contributors of *OpenEBS*. This article will focus on our strategies meant for the newbies trying to get their hands dirty with Go.\n\n***I have itemized our strategies as these steps:***\n\n1. Start your baby steps at GO [**Playground**](https://play.golang.org/). This will remove the initial hassles of downloading Go and setting up GOPATH & stimulate you towards coding (read *familiarising with the newer syntax*) & start believing the language by viewing & analyzing the output expected from your logic i.e. *WYSIWYG*.\n2. Get familiar with [**Vagrant**](https://www.vagrantup.com/) (*I understand this is a different tool & is in no way related to Go programming*). Vagrant with its pitch for ‘*Development Environments Made Easy*’ will help you to set up VMs which has all the necessary Go based downloads and GOPATH settings. One can search for simple open-source projects that does below:\n\n**a**.Vagrantfile that makes use of some sample Go programs (probably scripting a *git clone to *), &\n\n**b**. Makefiles that have the logic to compile & build.\n\nThere is one catch though! One has to install Vagrant & a preferred Hypervisor to enjoy the benefits of Step 2. So there is a bit of a learning curve involved.\n\n***However, this should not deter one to avoid Step 2, as it has huge benefits going forward. You will appreciate this step once you understand the difficulties in managing projects with their right versions and dependencies. A trivial mistake here will lead to bug injections. This step will rather help you enjoy the taste of CI right from your laptop.***\n\nAfter gathering enough confidence via these steps one can try out fancier stuff. Does ***running the builds in Travis*** or ***compiling the source code in a Docker container*** challenge you. You might as well explore some of this stuff & others in our [**tool-room**](https://github.com/openebs).\n\nThere may be various other options (*probably even simpler ones*) available in this '**DevOps**' world that are not mentioned here. E.g. [**GVM**](https://github.com/moovweb/gvm). However, I have tried to list down the game plans that have worked well for us. Do get back with your valuable insights on how you have succeeded to tame the initial apprehensions while trying out a new language.\n","notHasFeatureImage":true,"slug":"are-you-afraid-of-go?"},{"id":124,"title":"Tech Enthusiasts weighing in on Container Native Storage","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"27-07-2017","tags":["Docker","Game of Thrones","OpenEBS","Reddit","Statefulcontainers"],"excerpt":"These redditers are like the nights-watch-men (Operations Team) who are guarding the wall (production services) from army of dead (issues).","content":"\n*Full disclosure: I contribute to OpenEBS and I relish GoT (not IoT).*\n\n*This post is inspired by the *[*Reddit thread (r/Docker)*](https://www.reddit.com/r/docker/comments/6l0y3v/persistent_storage_with_docker_in_production/)* discussing implementation approaches for Stateful Containers.*\n\nThese redditers are like the nights-watch-men (Operations Team) who are guarding the wall (production services) from army of dead (issues). These redditers are akin to “the tech Enthusiasts” as depicted in the following diagram, who can foresee the challenges and are looking for innovative solutions. \n\n![Container native storage adaption curve](https://cdn-images-1.medium.com/max/800/1*11EOWUuoRjWn8pZ1uMXidg.png)\n\nWith S-less (server and state) architectures on the rise and faster networks at every possible endpoint, we can envision building and deploying services that can scale to yet unseen/unknown/unimaginable magnitude, **but!** (*Ned Stark once told me, that anything said before but*, ..*a GoT in-joke*) **State will be the bottleneck.**\n\nWhatever you do with Compute (S-less) containers, you always need to start and end with Data. Data is State. *State — is the beginning and end! the Alpha and Omega!*\n\nS-less will have no (busine$$) value without State! (*Well, agreed that you can offload the maintenance of State to someone else. But at what cost! Whoever owns the data, owns the $$*)\n\nThe real question is where to put the State!\n\n*Probably over-simplifying, but we can save the State using one of the following:*\n\n- *Connected Storage — Further divided into File/Block*\n- *Container Native Storage*\n\n**Connected Storage**: save the State external to container hosts and use the Orchestrator Volume Plugins to attach/detach storage from SAN/NAS or Cloud Disks.\n\nThere is a huge community (of cash rich vendors, a.k.a Roses, and Lions!) and a majority of enterprises (consumers of storage) locked in with these vendors are working on making this option succeed.\n\nThese enterprises are stuck with Roses or Lions not because they love them, but these are the only options that are available to them. This reverberates in the comments of the Reddit thread, here is a gist of the views expressed:\n\n(a) Connecting to NFS or EFS (from AWS)\n\nApplied Use Cases:\n\n- Applications that store data in the file — images or text or backup\n- Applications that need shared access to the data from multiple containers, running on different hosts.\n\nConcerns:\n\n- Hard to keep up with the performance and capacity needs of the massively scalable applications\n- Seeing Performance Issues — when saving a huge number of small-sized files\n- Need to be careful about Data Integrity from shared access. Needs a lot of hand-holding for secured access\n- Not a good option for databases or high random write workloads\n\n(b) Connecting to SAN or Cloud Disks (attached locally)\n\nApplied Use Cases:\n\n- Applications that are resilient to underlying storage failures like ElasticSearch/Cassandra\n- Databases like Percona/PostgreSQL\n\nConcerns:\n\n- Longer re-build times and degraded windows, as the size of the data increases.\n- Connecting one LUN per Container increases the boot-up times up to 10 minutes in worst cases\n\n*Roses and Lions have been there for a very long time, fighting for the dominion of the realms, by any means necessary. But these are summer lands. They are not prepared for the Great Winter that has begun in the North.*\n\nThe “tech Enthusiasts” (or the nights watch) have seen the challenges that the massively scaled applications (Winter) can bring and are seeking alternate options.\n\nA handful of vendors like [portworx](https://t.co/Aawo9fr4Dz), [storage_os](https://storageos.com/), [rook](https://rook.io/), [openebs](https://www.openebs.io/) are working on alternate options, what is now being termed as [Cloud Native Storage](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing) or [Container Native Storage](https://storageos.com/storageos-vision-cloud-native-storage-todays-modern/).\n\n*While the blogs above (and many others) dwell into what makes a storage container-native, the one that stands out is — ***the flexibility***.*\n\n*The tech enthusiasts/operations personnel will have the choice of technology used to deliver their services like Kubernetes, DockerSwarm, Mesos — on Google, Amazon, Azure, or Private Cloud with storage that integrates seamlessly with these cloud environments.*\n\n**Container Native Storage:** storage controller functionality is containerized and can co-exist with the containers (even fly with them) across the clouds.\n\nPortworx is leading the pack, has been successful in getting some reference customers (*we are yet to hear from them in the open forums — hard to convince the council at kings landing with just a reference!*)\n\nSome of the apprehensions surrounding this option are:\n\n- writing a new storage layer is hard.\n- missing some standard benchmarking tools that can clearly demonstrate the performance boost obtained by this relatively new way of provisioning storage.\n- there are questions raised about Rook / CEPH performance for DBs\n- dependency on the kernel drivers (this is probably hinting at PortWorx, if I read, between the lines/comments).\n\n*Winter is Here My Lord — We need new alliances!* We need to hear from more Operations and DevOps Personnel grappling with the storage issues.\n\nWe need them to spend more time towards sharpening the solutions that are being built for containers using containers in Open Source!\n\nDo contribute and earn your Open Source Karma by weighing in your thoughts at this *[Reddit thread (r/Docker)](https://www.reddit.com/r/docker/comments/6l0y3v/persistent_storage_with_docker_in_production/) or at [OpenEBS Slack](http://slack.openebs.io/) or at the [GitHub/Container-Storage-Interface](https://github.com/container-storage-interface/spec)*\n","notHasFeatureImage":true,"slug":"tech-enthusiasts-weighing-in-on-container-native-storage"},{"id":125,"title":"OpenEBS building Go Storage Kit Project — Maya","author":"Satyam Zode","author_info":"Go Developer @openebs | Open Source Contributor | Avid Learner","date":"25-07-2017","tags":["Golang","OpenEBS","DevOps","Container Orchestration"],"excerpt":"I attended GopherCon India 2017, there was a talk on “Package Oriented Design In Go” by William Kennedy. In that talk, William explained some really important and thoughtful design principles which we can apply in our day-to-day life, while writing Go.","content":"\n## Motivation\n\nI attended [GopherCon India](http://www.gophercon.in/) 2017, there was a [talk](https://youtu.be/spKM5CyBwJA?list=PLFjrjdmBd0CoclkJ_JdBET5fzz4u0SELZ) on “Package Oriented Design In Go” by [William Kennedy](https://twitter.com/goinggodotnet). In that talk, William explained some really important and thoughtful design principles which we can apply in our day-to-day life, while writing [Go](https://golang.org/project/). I have attempted to absorb some of the design philosophies I learned at GopherCon into practice at [OpenEBS](https://github.com/openebs). At [OpenEBS](https://github.com/openebs), as an open-source and growing Go project, We value Go principles and We try hard to leverage Go’s offerings.\n\nBriefly, OpenEBS is a [container-native storage](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing) that is built from containers for enabling stateful containers to get into production. Being container-native, OpenEBS augments the Container Orchestration (CO) Layers like Kubernetes, DockerSwarm, Mesos, etc., with Storage Specific Orchestration capabilities using “Maya” — Magic! I contribute mainly towards the “Maya” — which is a set of containerized control plane applications for hooking into several modules like the configuration, monitoring, and alerting of CO.\n\n## What is the Go Kit Project?\n\nTo understand in plain terms, let us take an example where we end up writing the same Go packages, again and again, to do the same task at different levels in the different Go projects under the same organization. We are all familiar with the custom logger package in the different Go projects.\n\nWhat if, the custom logger package is the same across the organization and can be reused by simply importing it, then this custom logger package is the perfect fit for the Kit project. The advantages of this approach go beyond avoiding duplicate code, improved readability of the projects in an organization, to savings in terms of time and cost as well :-)\n\nIf you go through Bill’s talk, you will notice that the Kit project is characterized by Usability, Purpose, and Portability. In this blog, I will discuss how I have applied the refactored code to use the “Kit Project” pattern for maya.\n\n## How to convert existing projects to have “kit”\n\nOpenEBS being a container-native project is delivered via a set of containers. For instance, with OpenEBS 0.3 release we have the following active maya related projects:\n\n1. openebs/maya aka ****maya-cli**** : is the command-line interface like kubectl for interacting with maya services for performing storage operations.\n2. openebs/mayaserver : or ****m-apiserver**** abstracts a generic volume API that can be used to provision OpenEBS Disks using containers launched using the CO like K8s, nomad, etc.,\n3. openebs/****openebs-k8s-provisioner**** : is the K8s controller for dynamically creating OpenEBS PVs\n\nWith these projects, we are already seeing how code gets duplicated when each of these projects is independently developed. For example *maya-cli* and *openebs-k8s-provisioner* both need to interact with *maya-apiserver*, which resulted in maya-apiserver-client code being written in *maya-cli* and *openebs-k8s-provisioner*. Similarly, *openebs-k8s-provisioner* and *maya-apiserver* have duplicated code w.r.t to accessing the K8s services.\n\nTo avoid this duplicity of code using the kit project, we are transforming OpenEBS/maya into a Kit project for the Application projects like [maya-apiserver](https://github.com/openebs/mayaserver), openebs-k8s-provisioner, and many more coming up in the future. openebs/maya contains all the kubernetes & nomad APIs, common utilities, etc. needed for the development of maya-apiserver and maya-storage-bot. In the near future, we are trying to push our custom libraries to maya. So that, it will become a promising Go kit project for the OpenEBS community.\n\nLets now see, how maya (as a kit project) adheres to the package oriented design principles:\n\n- ****Usability**** \nWe moved common packages such as orchprovider, types, pkg to maya from maya-apiserver. These packages are very generic and can be used in most of the Go projects in the OpenEBS organization. Brief details about new packages in Maya.\n1. Orchprovider: orchprovider contains packages of different orchestrators such as Kubernetes and nomad.\n2. types: types provide all the generic types related to orchestrator.\n3. pkg: pkg contains packages like nethelper, util, etc.\n4. volumes: volumes contain packages related to volume provisioner and profiles.\n- ****Purpose**** \nWhile the Packages in the Kit project are categorized as per the functionality, the naming convention should ideally provide the reader with the information on what the package “provides”. So, the packages (in the kit project) must provide, not contain. In maya, we have packages like types, orchprovider, volumes, etc. name of these packages suggests the functionality provided by them.\n- ****Portability**** \nPortability is an important factor for packages in the kit project. Hence, we are making maya in such a way that it will be easy to import and use in any Go project. Packages in the Maya are not a single point of dependency and all the packages are independent of each other. For example, types directory contains versioned Kubernetes and Nomad packages. These packages are simply importable to any project to use Kubernetes and Nomad APIs.\n\n## Example usage of maya kit project\n\nMaya-apiserver uses maya as a Kit project. Maya-apiserver exposes OpenEBS operations in form of REST APIs. This allows multiple clients e.g. volume-related plugins to consume OpenEBS storage operations exposed by Maya API server. Maya-apiserver will use volume provisioner as well as orchestration provider modules from Maya. Maya-apiserver will always have HTTP endpoints to do OpenEBS operations.\n\nSimilarly, openebs-k8s-provisioner will use the maya-kit project Kubernetes API to query for details about the Storage Classes, etc.,\n\nAnother usage is of the maya-kit project, maya-apiserver client that is accessed by maya-cli as well as the openebs-k8s-provisioner to talk to maya-apiserver.\n\n## Conclusion\n\nGo Kit project should contain packages that are usable, purposeful, and portable. Go Kit projects will improve the efficiency of the organization at both human and code levels.\n","notHasFeatureImage":true,"slug":"openebs-building-go-storage-kit-project-—-maya"},{"id":126,"title":"Cloud Native storage vs. marketers doing Cloud Washing","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"18-07-2017","tags":["Cloud Native","DevOps","Docker","Kubernetes","OpenEBS"],"excerpt":"Let’s try to protect the phrases Cloud-Native storage and Container Native storage; or maybe we should start using a yet more specific phrase such as Containerized Storage for Containers","content":"\nSome years ago, back when I was founding CEO of Nexenta during our high growth days, I tried to bring some rigor to the discussion around “software-defined storage” by setting out a definition. My basic point was that it wasn’t enough for storage to have APIs of some sort and hence to be able to be controlled by software -> in addition the storage ought itself to *be software*, and hence be able to be provisioned as software and so forth.*1*\n\nI wrote those blogs setting out a definition for software-defined storage because I was disgusted by the number of legacy storage vendors that went from denying that software-defined storage was a thing to, almost overnight, claiming that they *already did* software-defined storage. The risk was that the large sales and marketing budgets of legacy vendors would drown out the innovative companies that were actually building software that delivered on the benefits of software-defined storage thanks to, you know, *actually being software-defined*.\n\nFast forward 6 or 7 years and — **here we go again!**\n\nThis time we are seeing “Cloud Native storage” as a term being abused along with “Container Native storage.” Logically Cloud Native storage should be storage software that itself is, you know, *Cloud Native*. And that term is defined by people far smarter than me, however typically it includes some sense of microservices, and [12-factor approaches](https://12factor.net/), and the ability to consume cloud services. One great and lengthy definition of the term is explained by the inestimable Joe Beda [here](https://blog.heptio.com/cloud-native-part-1-definition-716ed30e9193).\n\nWe are even seeing Cloud Native storage applied to *any storage that serves Cloud Native applications*. Thanks to the work of Docker and Kubernetes and others, nearly any storage *can* serve storage for containers — albeit in a way that typically requires the user to change the way that they run the containers that are attached to the underlying storage to such an extent that many of the benefits of containerization are lost; I talk more about benefits of truly container-native storage below. Calling hardware based storage that ties into a Kubernetes environment via plug-ins “Cloud Native storage” is absurd and yet it is starting to happen. Such storage is no more Cloud Native than any legacy monolithic application is “Cloud Native” just because you’ve been able to cram it onto a container somehow.\n\nSpeaking of cramming monolithic apps into containers, if you have a famously difficult to manage monolithic storage solution that you cram onto a container or two — even if you integrate it nicely into Kubernetes or other Cloud Native environments — what you have maybe useful however it is NOT Cloud Native. Because, you know, it predates 12 factor and cloud-native approaches to building software by many years and, as such, is no more Cloud Native than ye old SAP or other traditional n tier app.\n\nAhh, that feels better.\n\nBut you might ask — *so what*? What’s the big deal about marketers abusing the English language for their own ends and distorting the definition of Cloud Native storage?\n\nWell, that comes down to whether we believe that the benefits of Cloud Native applications could apply to building storage software in a cloud-native manner. Is what’s good for the goose, good for the gander? Is Cloud Native storage even a term worth fighting over? Could a truly Cloud Native storage solution deliver benefits much as other Cloud Native software does?\n\n![Picture showing gooses](/images/blog/goose.png)\n\n**Let’s all be cloud native like the whale: goose, gander, OpenEBS mules, and friends**\n\nWell, why wouldn’t benefits such as:\n\n- better resiliency\n- better resource utilization\n- innate horizontal scalability\n- ease of troubleshooting\n\nand other benefits that apply to decomposed microservice based applications (aka “Cloud Native”) apply to software that is delivering storage?\n\nPerhaps just as importantly, wouldn’t the teams and communities that are building the software be able to be better organized — themselves loosely coupled and able to *DevOps on* — than those that are working on monolithic software? [Conway’s Law ](https://en.wikipedia.org/wiki/Conway%27s_law)does seem to be a major reason for the success of Cloud Native and microservices.\n\nThe benefit of that better team organization in my experience can be massively greater developer productivity. As many have pointed out, the productivity of developers in high performing DevOps / Microservices organizations is far, far superior; these developers deliver usable code 10–100x more quickly than developers stuck building to long release cycles with massive dependencies.\n\nSo, in short, here is my request of you. Just as we have been so careful in (not entirely successfully) protecting the term DevOps, let’s try to protect the phrases Cloud Native storage and Container Native storage; or maybe we should start using a yet more specific phrase such as ***Containerized Storage for Containers*** (warning, that’s a term we coined at OpenEBS). Otherwise, we risk polluting everyone’s efforts for a better way to build and run software in a Cloud Native way with marketing BS.\n\n1. (note — as I’ve tried to [explain elsewhere](/blog/software-defined-storage-finally), much of what we hoped to achieve with software-defined storage is only now being enabled, largely thanks to Kubernetes and other open orchestrators and containers as well).\n","notHasFeatureImage":false,"slug":"cloud-native-storage-vs.-marketers-doing-cloud-washing"},{"id":127,"title":"Container Native Storage builds its High Availability in style","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"17-07-2017","tags":["Docker","High Availability","Kubernetes","OpenEBS","Storage"],"excerpt":"Infrastructure components are the toughest to build. It is meant to work always by default.","content":"\nInfrastructure components are the toughest to build. It is meant to work always by default. It must be telecom grade as scores of applications rely on the infra component’s zero downtime, no deterioration in QoS, etc. However, we all know software has got defects and it is no different for infrastructure software components as well. One of the things the software can do is implement high availability as its core feature and somehow buy time (by the virtue of HA) when one of these components experiences a breakdown due to defects or otherwise.\n\nImplementing the HA piece seems to be the elixir to all the unforeseen issues. Is that so simple then? Programmers who have built the HA in the past will swear to its complexity. It is another piece of software logic that is very difficult to get right the first time or even after a couple of major releases.\n\n> The non HA programmer can have the flexibility to err. Well, a programmer is supposed to be a human after all. However, this option is ruled out for the HA logic implementer.\n\nHaving said this, implementing high availability logic in a '*Container Native*' software solution has reasons to cheer. Most of the container orchestration platforms have abstracted the scheduling, placements, evacuations, prioritized jobs, and whatnot and thus have freed the average programmer from these overwhelming tasks.\n\nAll of a sudden, programming becomes fun again (*HA logic becomes more like placing the [lego](https://en.wikipedia.org/wiki/Lego) blocks in a fashion/pattern to get your favorite character into action*).\n\nProgrammers now need to build the required HA logic by tying together appropriate placement components that result in a customized HA solution. We at OpenEBS [***toolroom***](https://github.com/openebs/) have been trying to maximize the offerings provided by [*Kubernetes*](https://kubernetes.io/) towards achieving a true container-native solution for OpenEBS high availability. All the advancements in Kubernetes will make OpenEBS better and the scenarios handled by OpenEBS (*which will never be simple*) can become a point of reference in Kubernetes.\n\nLet’s get into some storage specific HA basics:\n\n1. A typical persistent storage solution would require its data to be replicated across hosts within a cluster. There are also cases, where the production scenarios demand this replication to be across geographies. In addition, hybrid cases require some replicas to reside within the cluster while other replicas to be placed across zones, regions, etc.\n2. Since storage solutions derive their power from their underlying hardware resources, it makes sense to earmark exclusive hardware for storage components. In other words, the host(s) would want to avoid software components that are not storage specific.\n3. Solutions around placements alone are not sufficient for storage software to be highly available. There can be cases where a storage controller does not support active-active mode. Now relying just on placements can not avoid application outages due to these storage hiccups (*i.e. storage protocol connection breaks resulting in breakdown of the applications consuming this storage*). The evacuation should be fast & its new placement cannot take any time longer. Appropriate policies should be in place to let these storage components be evacuated early before the application components. This becomes more essential during a node crash where every component will fight for survival.\n4. Storage HA cannot rely on half-baked evacuations that may lead to split-brain conditions. Storage should either be evacuated or be let to die along with the crash. There should be no cases of hangs, stalls, freezes, you name it. There can be only one option & that is:\n\n***100% successful evacuation as well as 100% successful re-scheduling.***\n\nIn some cases, 100% successful evacuation might imply the use of the logic called [**stonith**](https://en.wikipedia.org/wiki/STONITH) before boarding the re-schedule flight.\n\nWith the above HA basics (*you may like to call it hardships*) in our mind let us find if the most popular container orchestrator has any feature, policy, etc that can simplify HA in storage. While looking at Kubernetes (version 1.7) I could find a couple of policies that can help storage build its HA story with ease.\n\n> One: ‘Node affinity’ is a feature that constrains which nodes your pod is eligible to schedule on, based on labels on the node.\n\n> Two: ‘Inter-pod affinity and anti-affinity’ is a feature to constrain which nodes your pod is eligible to schedule on based on labels on pods that are already running on the node rather than based on labels on nodes.\n\n> Three: ‘Taints and tolerations’ work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.\n\nDoes the above sound familiar? It does not introduce any new jargon except for the term 'Pod' and perhaps 'Labels'. A pod is a logical concept for a container or bunch of containers while labels can be understood as tags. Now if our persistent storage is really a piece of software that runs from within a container all the properties that Kubernetes exposes can be applied against this storage software & let it construct (or extend or even plug) its 'High Availability' feature.\n","notHasFeatureImage":true,"slug":"container-native-storage-builds-its-high-availability-in-style"},{"id":128,"title":"Test Driven Development — The DevOps Way","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"13-07-2017","tags":["Docker","Kubernetes","OpenEBS","TDD","Software Development","DevOps"],"excerpt":"TDD is the abbreviated form for `Test Driven Development`. It might also be true in-case of newbies to have never heard of TDD in the current season where DevOps, NoOps, DataOps, ML, IOT rule the roost.","content":"\nWe might have all heard of TDD, don’t we? TDD is the abbreviated form for 'Test Driven Development'. It might also be true in case of newbies to have never heard of TDD in the current season where DevOps, NoOps, DataOps, ML, IOT rule the roost.\n\nTDD in simpler terms would be planning and implementing test logic before implementing its development logic. However, I find it very difficult to do this in practice. What I do instead is a little bit of planning & start building the development logic. In the next iteration, I would start writing corresponding test logic i.e. unit test programming (*remember those jUnit days*).\n\nReaders may still be wondering how “*DevOps is related to TDD*”. Well, here it is:\n\n> *How about using the best of DevOps tools towards TDD implementation?*\n\nThe logic behind this is no brainier.\n\n> I find the very best of the open-source community busy in building & tuning tools old & new that are meant to satisfy DevOps culture & practices.\n\nIt makes sense for us to use these tools that meet our TDD needs.\n\nThe question that might be lingering now would be about DevOps vis-a-vis TDD. ***Is TDD not part of DevOps culture***? The answer is an emphatic yes. However, I am trying to pick up one thread at-a-time from the DevOps culture/practice. That thread here is TDD & is the focus area for this article. I would like to emphasize that TDD does not necessarily mean unit testing. It can be thought of as applying test logic, scenarios, etc as the team continues with its development efforts, that in-turn brings in changes, discoveries, ideas in test as well as in development. ***To sum it up, it tries to remove the pain associated with latter stage design changes*** In addition, the team is ***no more afraid*** of new features, bug fixes, requirement changes, etc. due to the presence of test logic that gets triggered due to any of these changes.\n\nHope to talk again about some tangible implementations we are doing in [OpenEBS](https://openebs.io/blog). I would end the story here with a tip.\n\n> *The popular currency of current age DevOps’ tooling is Docker & Kubernetes.*\n","notHasFeatureImage":true,"slug":"test-driven-development-—-the-devops-way"},{"id":129,"title":"OpenEBS on the Growth Path! Releases 0.3","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-06-2017","tags":["Jupyter Notebook","Kubernetes","OpenEBS","Postgresql","Storage Containers"],"excerpt":"It gives me immense pleasure to watch this baby (OpenEBS) successfully cross another milestone with its 0.3 release.","content":"\nIt gives me immense pleasure to watch this baby (OpenEBS) successfully cross another milestone with its 0.3 release. OpenEBS 0.3 not only marks the delivery of additional functionality but also comes with an all-rounded growth.\n\nThanks to our growing community (as depicted in the diagram below), we have had great conversations with thought leaders in the container world, who are constantly looking for ways to simplify the usage of Containers and Container Orchestrators. We seem to have hit a sweet spot in terms of what we set out to deliver with OpenEBS, which is resonating loud and clear with anyone, who is venturing to run persistence workloads in containers.\n\n![OpenEBS community tracker](https://cdn-images-1.medium.com/max/800/1*wnMG__-zl8yO06f7AAJh5w.png) \n\nWhile we have seen steady growth in the followers/subscribers on our community channels, we have also seen steady growth in the contributors to the project and we are happy to announce that we have been able to expand our team members at CloudByte(from across the world), to work full-time on OpenEBS.\n\nSignificant functionality changes include:\n\n- OpenEBS Storage Containers (VSMs) are completely orchestrated and managed via Kubernetes.\n- VSMs seamlessly work with your preferred pod networking\n- OpenEBS supports dynamic provisioners like EBS and GCP\n- *maya*, the Storage CLI can be used to fetch granular IO usage statistics similar to your Amazon Cloud Watch\n\nAdditional changes can be looked up at our [OpenEBS Project Tracker](https://github.com/openebs/openebs/wiki/Project-Tracker).\n\nWe are very attentive towards making the onboarding experience as smooth and simple as possible. OpenEBS 0.3, fully integrates into Kubernetes, allowing you to use OpenEBS storage on your existing Kubernetes using the following simple commands:\n\n---\n\n`kubectl apply -f openebs-operator.yaml`\n`kubectl apply -f openebs-storageclasses.yaml`\n\n---\n\nThe YAML files can be downloaded from [here](https://github.com/openebs/openebs/tree/master/k8s). If you don’t have an existing Kubernetes cluster, you can easily set it up using our [vagrant box](/blog/multinode-kubernetes-16-cluster-provisioning-made-easy-using-sandbox-vagrant-box). Detailed instructions for running OpenEBS on GKE can be found [here](https://github.com/openebs/openebs/blob/master/k8s/hyperconverged/tutorial-configure-openebs-gke.md). You could also use our [Ansible Playbooks](https://github.com/openebs/openebs/blob/master/e2e/ansible/openebs-on-premise-deployment-guide.md) to setup Kubernetes cluster with OpenEBS storage on-premise on physical servers OR virtual machines.\n\nWe also have (with the help of the community) focused efforts in providing [examples](https://github.com/openebs/openebs/tree/master/k8s/demo) of persistence work-loads. In the current release, we are happy to provide the following:\n\n- Clustered PostgreSQL Setup (inspired from [CrunchyData](https://www.crunchydata.com/))\n- Containerized Jupyter Notebooks (inspired from [kensu.io](http://www.kensu.io/))\n\nThe growth, in terms of the distributed teams, comes with the challenges of collaboration and maintaining the quality of the commits. One of our focuses with OpenEBS release 0.3 has been to augment our repositories with the tools that keep a watch on the code and product quality — we are now fully integrated with Travis, Codecov, GoReport and have built Ansible Playbooks for running CI via Jenkins.\n\n---\n\nAs we accelerate towards our enterprise version of the product towards the end of this year, we are all excited about the immediate milestones w.r.t:\n\n- Flush-out the details of Storage Infrastructure as Code (single source of truth)\n- Integrating into Container Monitoring Frameworks\n- Providing a UI\n- Optimize for high performance storage and networking infra.\n- Provide additional examples of Persistence Workloads that make use of the Application aware intelligence provided by OpenEBS\n\nWe always love to hear from you and what you think are the biggest storage operations pain points that you want to see eliminated. Do drop into our *[Slack Channel](http://slack.openebs.io) or stay connected with us via our other channels [Twitter](https://twitter.com/openebs), [GitHub](https://github.com/openebs/openebs/)*.\n","notHasFeatureImage":true,"slug":"openebs-on-the-growth-path!-releases-0.3"},{"id":130,"title":"Storage infrastructure as code using OpenEBS","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"29-06-2017","tags":["Containerised Storage","DevOps","Kubernetes","OpenEBS","Stateful Applications"],"excerpt":"With a vision to become de-facto block storage choice for stateful container applications, OpenEBS brings out many unique features to the open source storage space.","content":"\nWith a vision to become de-facto block storage choice for stateful container applications, OpenEBS brings out many unique features to the open-source storage space. One of the unique features that I talk about in more depth in this blog is “how storage platform builders and storage platform users can manifest their requirements as code using OpenEBS”.\n\nWith all the agility in Container and DevOps space, we are moving towards the thinking — “using infrastructure components such as network and storage for the application development should not be external to application development”. What I mean by this is, just like you write the code for your applications, you should be able to write some code or configuration for obtaining the storage and network infrastructure and you should get it.\n\n#### The YAML: Infrastructure coding language\n\nIn the paradigm of infrastructure as code, YAML is becoming the default coding language. Thanks to the increasing usage of YAML config files by the DevOps eco-system, they are becoming the default developer language to write the infrastructure code for the application development.\n\nSo, let us see what are the areas in which storage as infrastructure has to be thought in terms of code? This can be answered using the angle of who interacts with storage or the user persona. The simple answer would be the application developer and storage platform engineer. Next, what are the challenges today for a developer of a stateful application and the developer or engineer of an underlying persistent storage platform? In short, the developer needs\n\n- Some amount of persistent storage\n- With guaranteed availability\n- With guaranteed latency\n\nThe developer needs to quantify the amount of storage and the latency levels, which is a challenge always. The developer always starts with a guesstimate and refines later. The guaranteed availability is a characteristic of the underlying storage platform, but the developer may still need to decide some aspects related to the availability such as “how many copies of the data is needed, how often backups are to be taken, etc”. All these decisions to be taken by the developer, bring in the need for storage expertise or knowledge to the application developer, which usually is an additional burden to the developer. Traditionally, these decisions are taken by storage administrators who are adjacent to the application developers, creating a gap between what is needed and what is available. As the application scales, this gap widens, the application fails. **Yes, because something went wrong with storage!!**\n\nThe solution to this well known and well understood problem is to move towards a system where the application developer auto-magically gets help from the underlying storage system and he/she can continue to express/code the storage needs through the YAML language. The “so-called” storage administrator in the enterprise is going to be helped with an intelligent-storage-robot working continuously to align the storage to the application needs. With OpenEBS, the “intelligent-storage-robot” comes as an inherent component of the storage platform. The storage administrator can scale the platform and manage it at ease and do the job very well.\n\nThe intelligent-storage-robot or what we call OpenEBS as “Storage bot” handles the tough job of automating the storage actions such as\n\n- Updating the storage catalogues for application developers\n- Work with the storage platform to identify the capacity change needs, migration needs, backup needs and coordinate with the application\n- Work with the application for scheduling application aware snapshots, capacity management, data migration changes, automated data recovery etc\n- Work with a centralized ML engine (if there is one) to upload the application behaviour or to learn about the application behaviour so that it becomes more intelligent\n\n![Storage bot helping app developer of stateful container application and storage admin of the platform](https://cdn-images-1.medium.com/max/800/1*3wWTPR7i1gAVagzBYlOBmg.png)\n(***Storage bot helping app developer of stateful container application and storage admin of the platform***)\n\nAs depicted above, the storage solution designer is replaced by a storage-Bot (code) which is more effective as it predicts and updates the parameters of the storage platform as well as the application in real-time.\n\n#### OpenEBS helps to manage storage needs with YAML language/code\n\nOpenEBS divides the storage configuration into two parts. One, that the storage platform engineer has to design/manage and the other that the application developer has to design/manage.\n\n**The platform YAML file**\n\nThe platform configuration is specific to OpenEBS and the initial version is available at -\n\n[https://github.com/openebs/openebs/blob/master/k8s/openebs-config.yaml](https://github.com/openebs/openebs/blob/master/k8s/openebs-config.yaml)\n\nThe storage platform engineer develops this YAML file to scale the storage platform from few nodes to thousands of nodes across multiple locations. The YAML files can be version-controlled in a git repository and deployed into production to roll out the storage build-up process.\n\n![Storage platform deployment through YAML file automation](https://cdn-images-1.medium.com/max/800/1*WiZS5A4iLPeMtwxUmdxunQ.png)\n(***Storage platform deployment through YAML file automation***)\n\nA real example of this automation would be something like —\n\n---\n\n*the storage bot senses that there is a flood of large block size reads about to hit on a postgres-volume-246 which has its copies on node2 and node4 and to satisfy the latency needs of this volume, the read-cache on node4 is insufficient and one of the spare SSD on node4 can be re-purposed as read-cache temporarily. Storage bot learns this situation, makes the decision to do so and modifies the openebs-config.yaml on node4 as a new git version and rolls out the config file. Once the workloads are processed successfully after a few hours/days, the read-cache SSD is re-purposed back by the storage-bot either by updating the config again or by rolling back the config change.*\n\n---\n\nThe above is a very simple example and more complex things like “automated data migration to a correct node or correct tier, adding compute and memory resources to the storage controller nodes on the fly, automated software upgrades, etc” can all be achieved by the storage-bot using the git-versioned openebs-config.yaml files\n\n**The application YAML file**\n\nNow to the application developers paradise 🙂\n\nOpenEBS readily provides the storage configuration templates/storage catalogs for various stateful applications and application variants. The storage classes will be an ever-growing list with more participation from the community, but the initial version is at — [OpenEBS Storage Classes](https://github.com/openebs/openebs/blob/master/k8s/openebs-storageclasses.yaml)\n\nThe storage class will help the application developer with various storage config parameters. Some of the possibilities are:\n\n— Initial capacity, capacity growth rate per week/month, latency expectations, active data size (hot/cold data ratio), backup schedules, application data patterns, number of copies to be saved, typical instructions during upgrades, data migration boundary conditions, etc\n\nAs you can see, the above actions form a major part of the job of storage administrator who plans, designs, and executes the storage parameters. In the case of OpenEBS, the application developer can code these requirements into the application YAML file and the OpenEBS storage bot will enforce and help continuously optimize this file as well. This, I believe is a huge step forward in the journey of simplifying the storage infrastructure interface to the application developers.\n\nOpenEBS is in the early days. With the release of OpenEBS 0.3, the DevOps users have an option to get the container-native block storage for their stateful workloads orchestrated by Kubernetes. OpenEBS 0.3 release is the first step in making the storage infrastructure as code possible.\n\nLooking forward to receiving feedback/comments/criticism on this topic. We are lurking on slack at [https://openebs-community.slack.com/](https://openebs-community.slack.com/)\n","notHasFeatureImage":true,"slug":"storage-infrastructure-as-code-using-openebs"},{"id":131,"title":"Storage is Dead! Long live OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"28-06-2017","tags":["DevOps","Kubernetes"],"excerpt":"In this blog I share something I’ve been saying for a while now that people suggested might just be audacious enough to spark a conversation.","content":"\nIn this blog, I share something I’ve been saying for a while now that people suggested might just be audacious enough to spark a conversation.\n\nPlus — I explain why I’ve decided to help found another storage project — [OpenEBS](http://www.openebs.io/) — and to become CEO of the underlying company, [CloudByte](http://www.cloudbyte.com/), at a time in which venture capital funding to space has declined (at least if you accept Rubrik and [Cloudian](https://cloudian.com/) and similar) funding as outliers :)).\n\nTL;DR: the old storage industry is ****dying**** like many stagnant and declining IT sectors before it, killed through outdated business practices and architectures and the emergence of a new set of personas that have radically different requirements, expectations, and even tastes. The new architectures and business models already exist — though they are unevenly distributed.\n\nI lay the story out in three points. I keep it brief and link to underlying technical resources w/ code and demos via those links for those that are interested. Please discuss below and/or at: [@epowell101](http://twitter.com/epowell101). Better yet, click here to be invited to our Slack channel: [https://openebsslacksignup.herokuapp.com](https://openebsslacksignup.herokuapp.com/)\n\n****Storage is dead? How so?****\n\n*On the one hand, massive data growth continues.* It is easy to look at a graph like the one above showing ongoing exponential growth in data created and conclude that the funding winter that has hit storage is just another irrationality of the venture market. Perhaps it is like 2006–2008 when too many of the top enterprise technology VCs had a couple of walking dead storage companies in their portfolio and wouldn’t or couldn’t take on new investments.\n\nAnd many thanks to Mary Meeker and Kleiner Perkins for her [annual report](http://www.kpcb.com/internet-trends) on trends for the graph.\n\n****Cloud****\n\n*On the other hand, cloud.* The major cloud vendors are a) truly major and b) see storage as a way to control customers.\n\nTo the ‘truly major’ point — the three largest cloud providers are some of the largest companies the world has ever seen. Those of us that remember the power of WinTel in the 90s and early 2000s need to remember that this time is different in part because the companies are massively more wealthy thanks in large part to having business models that transcend technology.\n\nTo the ‘control customers’ point — data gravity exists. And it is further leveraged by cloud providers making it free to transfer data to them and rather expensive to get it back.\n\nWhich brings us to DevOps and cross-cloud services…\n\n****DevOps and cross-cloud services****\n\nAs I learned from my experience as a co-founder and CEO of [StackStorm](https://github.com/StackStorm/st2), DevOps is a radically better way of building and running software — as well as being a cultural movement.\n\nAnd while there are many, many characteristics of what constitutes a “DevOps environment” — one important aspect as I see it is that the intent of the developer can be fully manifested by the underlying infrastructure and related services.\n\nAnd a second aspect is that many capabilities are consumed by the applications and the application developers as services. This fits with the use of microservices as well, of course.\n\nLast but not least, DevOps is a culture and a set of practices that, as demonstrated by the story of the crucial book the Phoenix Project, is motivated in part by a realization of how broken, slow, dysfunctional, unfair, and brittle, proprietary IT systems and IT operations traditionally have been. From this perspective, applying infrastructure as code in part to ensure cross team transparency and blameless postmortems and so forth are not just examples of useful techniques — they are manifestations of the values of a better culture.\n\nOne strength of clouds like AWS is that they enable you to consume storage in a DevOps friendly way — your orchestration or other automation can deal with the provisioning and management (or can rely on the solutions provided by the cloud provider) so that intent can flow from the application through the infrastructure; in addition, the infrastructure is there to be consumed dynamically, via APIs. Unfortunately, the cloud providers themselves have become the new proprietary, which makes some in the DevOps world a bit uncomfortable as in the past lock-in was used by proprietary vendors to maximize their own prices at the expense of their customers. Kubernetes especially offers a chance to keep the wiring free from the cloud providers, hence providing some hope of avoiding lock-in.\n\n****Uhh, huh — and OpenEBS?****\n\n![Uhh, huh — and OpenEBS?](/images/blog/uhh-huh-openebs.png)\n(***Uhh huh — data, cloud, DevOps, lock-in and, err, OpenEBS?***)\n\n\nIn short, the highest level vision for OpenEBS is one in which the DevOps friendly aspects of running storage in the cloud is now available on-premise and across your clouds, thus freeing you from vendor lock-in much in the way that Kubernetes helps you avoid the risk of locking yourself in through the use of cloud service specific wiring and workflow.\n\nWhat if your developers and DevOps teams could orchestrate your storage controllers just like other containers? How much more productive could they be?\n\nAnd what if that set of storage services allowed you to treat your stateful workloads on containers much like the ephemeral containers that have proliferated?\n\nAnd what if your storage and CIO teams were able to establish policies that governed your data without impeding the agility that was your purpose in moving towards containers and DevOps in the first place?\n\nLast but not least, what if, having tooled your environment to work with Amazon it just worked when you added the on-premise and cross-cloud OpenEBS? Wouldn’t this limit your lock-in with any one cloud vendor?\n\nOpenEBS remains early. Today we are at `release 0.3` and are just starting to be used by enterprises like Cap Gemini and others to deliver storage for stateful workloads on containers.\n\nI’m back into the storage industry because I’m sure that OpenEBS has a good chance to allow storage to slip into the background and “just work” to support the incredible boosts in agility delivered by DevOps.\n\nA bunch of us imagined a better storage industry — one more software-like, one that eliminated vendor lock-in — back in the 2008 / 2009 time frame. Instead, we got some kick-ass flash and quite cool hyper-converged solutions PLUS, mostly, AWS just working (well enough) with a far better business model and ability to scale.\n\nSo vendor lock-in remained a problem. And storage far too often remains the bottleneck that slows down the good stuff, including the shift of real workloads onto containers.\n\n[OpenEBS](http://www.openebs.io/) is going to fix all of that and make storage and related storage services something you use to free yourself from lock-in while boosting your operational agility. Game on!\n\nYou can learn more about today’s release of OpenEBS 0.3 from our COO and co-founder’s blog [here](https://openebs.io/blog/author/uma-mukkara). I think you’ll agree, what’s possible now reinvents storage fundamentally — with your help we’ll turn this project into a company and a complete solution that we can all rely on.\n\nOther resources include demos of OpenEBS working seamlessly with Kubernetes and interesting stateful workloads including PostgreSQL and the Spark notebook Jupyter in our demo repo [here](https://github.com/openebs/openebs/tree/master/k8s/demo/crunchy-postgres).\n","notHasFeatureImage":false,"slug":"storage-is-dead!-long-live-openebs"},{"id":132,"title":"Data Scientists adopting tools and solutions that allow them to focus more on Data Science and less…","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-05-2017","tags":["Adalog","Data Science","Kubernetes","OpenEBS","Jupyter Notebook","Solutions"],"excerpt":"Data Science as we all know is becoming crucial to many if not most businesses around the world. Data Science and ML are decidedly, the most trendy skills that a lot of people are aspiring to acquire or be associated with.","content":"\nData Science as we all know is becoming crucial to many if not most businesses around the world. Data Science and ML are decidedly, the most trendy skills that a lot of people are aspiring to acquire or be associated with. In my recent interactions (in the context of hiring for OpenEBS) with college graduates and software engineers, the one question that pops up is, if they join OpenEBS, will they get a chance to work on ML / Data Science (and the answer is a definite “maybe” :)).\n\nThe lure towards “Data Science” has gone beyond the $$ paid to the Data Scientists. It is about the process of creating wonders. “Data Science” puts forth a unique mix of interesting challenges, that requires grappling with the enormous amounts of data and the mind numbing application of machine learning and computational algorithms, visualization, and soft skills. These aspects associated with “Data Science” are evolving, as depicted in this [data science Venn diagram](https://www.infoq.com/articles/christine-doig-data-science-team-discipline).\n\n![Data Science Venn Diagrams](https://cdn-images-1.medium.com/max/800/0*FEPtZ-3YF48YMkRs.)\n\nYou can see this trend expressed in this graph of searches on Google as well. An interesting point is that “Machine Learning” has been neck and neck with “Data Science” until recently when “Machine Learning” started to be searched approximately 30% more frequently. And the gap between searches about Data Science and about Data Scientists has widened in the last couple of years; it seems people are about 3x more likely to be interested in Data Science than in the Data Scientists that actually do the work 🙂\n\n![Graph of Machine Learning, Data Science, Data Scientist, and Deep Learning searches on Google](https://cdn-images-1.medium.com/max/800/0*Usx-pyQi3bHCTyJe.)\n\nInterestingly, we have seen early OpenEBS users and contributors asking about OpenEBS delivering storage to containers running distributing nodes for Kafka or Spark or other pieces of a Data Science pipeline. We wanted to learn more about the needs of these users.\n\nThankfully, we are a little bit of a sister company to [Kensu.io](http://www.kensu.io/) as our Chairman [Evan Powell](https://twitter.com/epowell101) has been advising [Kensu](https://twitter.com/kensuio) as well. So we got in touch with the Kensu.io founders [Andy](https://twitter.com/noootsab) and [Xavier](https://twitter.com/xtordoir), who are co-creators of the [spark-notebook](http://spark-notebook.io/), which is the leading Scala notebook for data scientists working in the Spark community. [Andy](https://twitter.com/noootsab) and [Xavier](https://twitter.com/xtordoir) and their team have built and have helped to run some of the largest Data Science pipelines in finance and related fields in Europe; plus they are O’Reilly authors. They have been really generous with their time.\n\nThe TL;DR is — *data scientists still spend way too much time hacking together Data Science pipelines and if they only could orchestrate the end to end pipeline with Kubernetes/Mesos (which seems to be common in these environments) so that storage itself worked simply, this could be more productive, could recreate their results more easily, and might even work together better.*\n\nIn the rest of the blog I set out a little more specifically what we have learned and we share the beginnings of a recipe for Data Science on containers with OpenEBS as the storage. Please provide feedback to this blog or join the now hundreds of people on the [OpenEBS slack channel](http://slack.openebs.io/) and post your comments and feedback there.\n\n**Data Science workflow hindered by infrastructure management tasks**\n\nThe workflow of a data science project typically comprises four stages as outlined below, with data scientists going back and forth between these stages, before arriving at product/insights in the form of a report, dashboard, or data service.\n\n![Data Science workflow](https://cdn-images-1.medium.com/max/800/0*UCJHD8p2bNOemyC9.)\n\nEach of these stages involves tasks that can be further classified into three distinct types depending on the nature of the task as follows:\n\nType 1: Data Science\n\nType 2: Best Practices or Governance\n\nType 3: Administer Infrastructure/Tools\n\nData Scientists are primarily interested in *Type 1: Data Science tasks*, some sample tasks under this category (at different stages of the workflow) are:\n\n![Data Science tasks at different stages of the workflow](https://cdn-images-1.medium.com/max/800/0*t41w0qHFapFBWtX_.)\n\n*One of the crucial requirements for the data scientist is to produce accurate and reproducible analysis by collaborating with larger teams. *[Martin Hack](https://twitter.com/mhackster)*, commented that he has come across situations where the accuracy of the models change from the time the models were generated to the time that they are presented to a wider audience. This is usually attributed to the change in the data using which the model was generated.*\n\nThe efficiency and confidence with which these tasks are performed or improved (in an iterative way) are dependent, in the current scenario, on some *best practices*. The best practices range from following certain naming conventions for the data files used to tuning infrastructure/tools — like databases, distributed systems that are used to run the analytical or modeling tasks.\n\nSome sample Type 2: Tasks at different phases of the Data Science workflow are as follows:\n\n![Tasks at different phases of the Data Science workflow](https://cdn-images-1.medium.com/max/800/0*IKPVzSqbyirvwZW5.)\n\nThe tasks mentioned above just provide a glimpse into the challenges associated with a typical data science workflow. In workflows that require creating models by using data from multiple live sources, tracking the accuracy of the data without having something like application error logs, becomes a harder problem to solve, if not impossible. This also leads to anxious data scientists who tend to keep their work private, unless a higher level of confidence is achieved on the models generated.\n\nTalking of confidence, today we take pride in writing code that goes into production on day one, using automated continuous integration systems. Kensu is doing something similar, which can be considered as *Continuous Integration mixed with Live Monitoring of Data Sources.*\n\nAndy and Xavier and the team are rolling out ***Adalog***, their software and SaaS solution, first for their existing professional service customers and then more broadly. While you can learn much more about Adalog on their website — [http://www.kensu.io](http://www.kensu.io) — as I understand its Adalog picks up all the relevant metadata of the data science pipeline and uses it, as well as the underlying systems themselves, to deliver compliance reporting and controls.\n\nData Science Notebooks (like [*Jupyter*](http://jupyter.org/), [Spark-Notebook](https://github.com/spark-notebook/spark-notebook), etc.,) and integrated products like Adalog have enabled improved and reproducible data since workflows, and deploying these notebooks towards containers have helped in boosting the productivity.\n\nYet, there are a few infrastructure related tasks that are not fully integrated into this workflow, which creates a manual dependency on the expertise of infrastructure administrators. At the same time, it also burdens the administrators/operations team to manage a new kind of workload in their environment.\n\nSome of the tasks that fall under this *Type 3: Infrastructure / Operations* tasks from (from a storage operations perspective) are:\n\n![Infrastructure / Operations tasks from (from a storage operations perspective)](https://cdn-images-1.medium.com/max/800/0*gcVdcir6RcbetPXe.)\n\n*Andy and Xavier stated that they have the integration pieces with the data science tools, he finds that storage infrastructure related challenges remain and many of these would seem to be best solved with tighter integration by-products like OpenEBS that are fully containerized and hyper coverged, with products like Adalog.*\n\n*Without a solution like OpenEBS that natively integrates into the orchestration and that containerizes the storage itself*\n\n- *Capacity management is very tricky, where some of the tasks involve non-deterministic data bloating. Hitting disk space errors may result in the loss of many hours of processing.*\n- *Managing the capacity for the local copies that need to be maintained for external data sources and keeping them in sync and distributing to team members can be very time consuming*\n- *The dynamic nature of jobs that require bursts of IOPS/Throughput means that to be safe the systems are massively over provisioned which then means you need a large budget request even to just mess around in trying out new models for example*\n- *Maintaining the provenance of the data along with the models for reproducing the results with the same accuracy they had during model generation is extremely difficult and without that the human workflow of many data science users is broken; imagine, for example, trying to compare the performance of multiple teams who are being evaluated based on the quality of their models if you cannot be sure that the training sets have remained trusted.*\n\n*Andy says, these infrastructure and hand-holding of the workflow operations are what makes Data Science more of engineering than science.*\n\n**Data Science workflow — storage management tasks automated by OpenEBS**\n\nDocker and Kubernetes have eased the operations around maintaining large scale computational clusters, and with containerized and hyper-converged storage like OpenEBS, the operations can now provide storage for millions of containers without having to maintain separate storage silos.\n\nKubernetes and OpenEBS will help in further simplifying the infrastructure management tasks for the data science workflows by supporting automated storage capacity management based on the data source changes, auto-tuning of the QoS depending on the priority of the analytical task, and provide data provenance without laying an extra burden on the data scientists by deeper integration with the notebooks.\n\n*For instance, the following is a glimpse of what can be achieved through deeper integration of Adalog with Kubernetes and OpenEBS:*\n\n1. *Data scientists starts a new notebook ( or project ). This project is instantiated on the container cluster (say Kubernetes).*\n2. *When a data source is specified for download, it would trigger provisioning of a new OpenEBS storage volume from this cluster for the data source. The capacity for this new volume is auto-managed by OpenEBS. Adalog would maintain the information of the volumes associated with a notebook.*\n3. *When the notebook finishes downloading the data, it will trigger to create a snapshot of the downloaded data. Notebook can allow for regular syncing with the source to check for updates and download the incremental data etc., Adalog would track the snapshots of the volume used by the notebook.*\n4. *As data scientists start exploring the data, there are provided with read-only access to the snapshot of the data-source that is already downloaded. And new storage volumes are created for storing the temporary files. Adalog can associate the volume created with the user who initiated the data generation.*\n5. *When a certain exploration is aborted or cleared from the notebook, the associated volume ( and all the temporary files) are also cleared from the storage. Adalog would keep track of the functions and models generated via the exploration phase along with metadata information about the results generated, even though the volumes (data) is deleted.*\n6. *OpenEBS volumes come with fine-grained allocation of resources and auto-tuning which can help with running high-priority modeling jobs faster than the others.*\n7. *From Adalog, the user can trigger OpenEBS volumes (data) to be backed up or restored from S3 bucket.*\n\nAs a next step, we are planning to provide a sandboxed version of adalog running on Kubernetes and storage managed by OpenEBS. Much like the vagrant box we built to enable OpenEBS to be tried out on Kubernetes, and then became a favorite tool of some largely for trying out Kubernetes — we are hopeful that this sandbox will be useful as well as being a way to see OpenEBS in action.\n\n*If you want to earn massive open-source karma (and stickers of course) — let us know on our *[*OpenEBS Slack channel*](http://slack.openebs.io/)* that you’d like to help with this solution building exercise.*\n","notHasFeatureImage":true,"slug":"data-scientists-adopting-tools-and-solutions-that-allow-them-to-focus-more-on-data-science-and-less…"},{"id":133,"title":"Multi-Node Kubernetes 1.6 Cluster provisioning made easy using SandBox (Vagrant box)","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"23-05-2017","tags":["Kubeadm","Kubernetes","OpenEBS","Vagrant","Virtualbox"],"excerpt":"Working on OpenEBS, a containerized storage for containers which is orchestrated by Kubernetes, most of our tasks, be it development, testing and demo require us to setup and modify the nodes in Kubernetes cluster.","content":"\n### Background\n\nWorking on OpenEBS, a containerized storage for containers which is orchestrated by Kubernetes, most of our tasks, be it development, testing and demo require us to setup and modify the nodes in Kubernetes cluster. In addition, a multi-node cluster is a must as we go beyond the initial development and testing, to explore the high availability, scale, performance, and upgrade aspects.\n\nWhile *minikube* and *minishift* provide an easy way to setup Kubernetes single node cluster — for multi-node cluster the fastest ways to get going are usually cloud or hosted solutions. ***kubeadm*** is the closest we can get to easily setup a cluster, but since it is still in alpha, we keep running into some issues like — [kubadm init v1.6.1 fails](https://github.com/kubernetes/kubeadm/issues/226)\n\n*An ideal solution for a developer would be a Kubernetes Sandbox. A sandbox that can be easily setup on a laptop and can work on the move (without net connectivity.) This Sandbox should be shielded from the different API or CLI changes that happen with the frequent releases of kubeadm and Kubernetes.*\n\nWe have used Vagrant, VirtualBox, and Atlas to do just that.\n\n![Vagrant, VirtualBox and Atlas](https://cdn-images-1.medium.com/max/800/1*7kkviZOwgh8ePDYRjFX0mQ.png)\n\n## Try It! It is Easy and Quick!\n\nOnce you have Vagrant (1.9.1 or higher) and VirtualBox (5.1.14 or higher) installed on your laptop/machine, just do the following:\n\nStep 1: Download the Vagrantfile from [OpenEBS Github](https://raw.githubusercontent.com/openebs/openebs/master/k8s/lib/vagrant/test/k8s/1.6/Vagrantfile)\nStep 2: Run **vagrant up**\n\nDetailed instructions can be found [here](https://github.com/openebs/openebs/tree/master/k8s/lib/vagrant/test/k8s/1.6).\n\nThe above two steps will provision the following:\n\n- Ubuntu VM with Kubernetes Master (kubemaster-01)\n- Ubuntu VM with Kubernetes Minion (kubeminion-01) associated with (kubemaster-01)\n- Setup *weave* as pod network\n- Setup the kubectl credentials ( admin.conf) on kubemaster-01\n- Sample Kubernetes pod YAML files are located on (kubemaster-01) under the directory (*~/demo/k8s/spec/*)\n\nIn addition to the above, the following OpenEBS provisioning tasks are also performed.\n\n- Install OpenEBS iSCSI FlexVolume Driver on the kubeminion-01\n- Ubuntu VMs installed with OpenEBS Maya Master and OpenEBS Storage Hosts. (If you don’t want to use the storage, you can skip the installation of these VMs. Check the customization steps below).\n\n## Customizing the Kubernetes Sandbox\n\nThe above instructions include setting up Kubernetes and OpenEBS as well, but you can easily customize the Vagrantfile to skip installation of OpenEBS by prefixing ENV variables before the vagrant command as follows:\n\n MM_NODES=0 MH_NODES=0 vagrant up\n\nSome of the configuration options available are:\n\n- KM_CPU — Number of CPUs for minion (*default 2*)\n- KM_MEM — Size of the RAM (in bytes) for minion (*default 2048*)\n- KH_NODES — Number of Kubernetes Minion VMs (*default 1*)\n- KH_CPU — Number of CPUs for minion (*default 2*)\n- KH_MEM— Size of the RAM (in bytes) for minion (*default 1024*)\n- MM_NODES — Number of OpenEBS Maya Master VMs (*default 1*)\n- MH_NODES — Number of OpenEBS Storage Host VMs (*default 2*)\n\nIf you are looking for an older release of Kubernetes, checkout — [kubernetes vagrant boxes with 1.5.5](https://blog.openebs.io/setting-up-kubernetes-1-5-5-cluster-with-vagrant-dda11e33b5bc)\n\n## Contributing to creating Kubernetes Sandboxes\n\nBtw, the process of creating these Kubernetes Sandboxes is Open Sourced.\n\nThe majority of the issues that are encountered during the Kubernetes cluster setup using kubeadm are related to the software api/cli options changed across different versions of kubeadm or the interfaces between kubeadm and Kubernetes. Another nagging issue is the need to have connectivity to the network.\n\nThese issues are resolved by having Sandbox (vagrant boxes) that pre-package the required software with versions that are compatible. The task of downloading the required software is automated via the scripts.\n\nOnce a VM is initialized with network/IP address details, certain initialization tasks will have to be executed. These are placed in the configuration scripts (which are also pre-packaged with the sandboxes) and are invoked from the Vagrantfile itself.\n\nCurrently, the Sandboxes use *weave* as a pod network, you can easily extend this to use a different scheme for pod network.\n\nIf you like to contribute or learn more about these box generation scripts, check out our [GitHub](https://github.com/openebs/openebs/tree/master/k8s/lib/vagrant) or join our [*Slack Channel*](http://slack.openebs.io).\n","notHasFeatureImage":true,"slug":"multi-node-kubernetes-1.6-cluster-provisioning-made-easy-using-sandbox-(vagrant-box)"},{"id":134,"title":"Rancher’s Longhorn announcement solidifies the OpenEBS story","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"18-04-2017","tags":["Docker","OpenEBS","Rancher","Stateful Applications","Longhorn"],"excerpt":"Today, Sheng Liang unveiled project Longhorn as a new way to build distributed block storage for cloud and container-based platform.","content":"\nToday, Sheng Liang [unveiled project Longhorn](http://rancher.com/microservices-block-storage/) as a new way to build distributed block storage for cloud and container-based platform. We, the OpenEBS team, are thrilled with this news as it solidifies our decision of using Longhorn as the underlying block storage system for OpenEBS.\n\nWhen I reviewed the OpenEBS vision with Sheng late last year, he introduced me to Longhorn project and asked me to study it and use it if it makes sense for OpenEBS. Clearly, Longhorn was sharing the OpenEBS block storage vision and we soon launched OpenEBS with Longhorn underneath.\n\nWhat I liked most about Longhorn was the nice separation of block protocol stack (Longhorn controller) and the actual block storage (Longhorn replica). This helped us containerize the storage software completely and efficiently. An OpenEBS VSM, or a storage pod, will have containerized longhorn controller and replica(s).\n\nAnother cool thing about Longhorn implementation is that it’s controller sits closer to the application docker containers thus enabling hyper-converged mode of deployment and it’s replica sits closer to the underlying storage with a simple sparse file based system for block data management.\n\nIt is so nice of Sheng to mention OpenEBS’s usage of LongHorn. We will continue to contribute to the performance tuning of LongHorn, improvements to S3 integration, rebuilding logic of replicas, etc\n\n### OpenEBS is an extension to the idea of Longhorn\n\nSheng Liang, in his blog, talks about Longhorn and other storage systems.\n\n“*We wrote Longhorn as an experiment to build distributed block storage using containers and microservices. Longhorn is not designed to compete with or replace existing storage software and storage systems…”*\n\nAs believers in the idea for which Longhorn is written, we are building OpenEBS as credible enterprise storage with Longhorn at it’s core. As part of the journey with CloudByte ElastiStor, where we have containerized the storage volumes a few years ago, we picked up tremendous amount of real world experience in delivering enterprise storage to the customers. All this experience is being put into work at OpenEBS. OpenEBS will have most of the enterprise features that Sheng mentions, if not more. Some of them will be “scalable and reliable distributed file systems, a unified storage experience, enterprise data management, crash consistency, near disk performance, etc”.\n\nAgain, a ton of thanks to the awesome folks who contributed to the development of Longhorn. With this announcement from Rancher, we are hoping to see more usage of Longhorn and OpenEBS. We are all ears to the community for any suggestions on OpenEBS and its usage of Longhorn ([slack.openebs.io](http://slack.openebs.io)).\n\nLong live Longhorn!!\n","notHasFeatureImage":false,"slug":"rancher’s-longhorn-announcement-solidifies-the-openebs-story"},{"id":135,"title":"OpenEBS sprinting ahead, 0.2 released","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"10-04-2017","tags":["Containerized Storage","Docker","Kubernetes","Microservices","OpenEBS"],"excerpt":"I am delighted that we have been able to push the OpenEBS 0.2 release for community consumption, a version that demonstrates that storage controllers can be containerized.","content":"\nI am delighted that we have been able to push the [OpenEBS 0.2 release](https://github.com/openebs/openebs/releases/tag/v0.2) for community consumption, a version that demonstrates that storage controllers can be containerized. This release comes with k8s flex volume driver called “openebs-iscsi”\n\nThe past week has been very exciting for the OpenEBS Team — engaging with container technology evangelists, enthusiasts, and users at the [Bangalore Container Conference — BCC 2017](http://www.containerconf.in/), followed by a very interactive meetup on the [Containerized Storage for Containers](http://www.containerconf.in/) with [Evan Powell](https://twitter.com/epowell101), [Ian Lewis](https://twitter.com/IanMLewis), and my fellow Bangalore Entrepreneurs and OpenSource fans.\n\nBy containerizing the storage functionality, the OpenEBS delivers the core storage capabilities like block-layout and its management, data protection, consistency, and availability as a “microservice”, thus bringing the advantages of containers to the storage volumes. Containerization also helps in dynamic provisioning at scale, scale-up/down the storage cluster, monitoring, ease of upgrades, etc.\n\nWe understand that generic orchestration engines will not solve all the problems of storage orchestration. We are augmenting container orchestration engines with the storage intelligence by the OpenEBS orchestration layer — maya. In line with this vision, the 0.2 release has implemented m-apiserver that acts as an interface to the Volume Plugin drivers to provision the storage, while it takes care of interacting with the Container Orchestration Engine to find the right placement for the Storage Containers (VSM).\n\nOpenEBS 0.2 can be used to setup an Amazon EBS-like Block Storage Service for your containers, and consume block storage for your Stateful applications.\n\n![OpenEBS 0.2 setup](https://cdn-images-1.medium.com/max/800/1*itiDxdwyTmdd9VsIYwFYiA.png)\n\nThe Ops team can easily setup an OpenEBS cluster using Bare Metal Machines or VMs, just like setting up Kubernetes Cluster. The Kubernetes minion nodes should be configured with the FlexVolume OpenEBS Driver (openebs-iscsi) to use OpenEBS Storage.\n\nThe DevOps or the Developers can configure Storage to their Stateful Apps from OpenEBS cluster by having their Application intent files point to OpenEBS Maya API Server.\n\nOpenEBS is a completely OpenSource Project that is being driven by the feedback received from the community. In our next milestone 0.3, we are working towards making OpenEBS hyper converged with Kubernetes. Enhance the capabilities around Core Storage for using an additional type of disks for storing block data, providing the controls to the user to perform backup/restore to Amazon S3, etc., Do take a moment to check out our [Project Tracker](https://github.com/openebs/openebs/wiki/Project-Tracker).\n\nIf you are as cautious and skeptical about technology as me, then seeing is believing. To help you quickly get started, we have created Kubernetes and OpenEBS 0.2 Vagrant Boxes.\n\n[Try OpenEBS 0.2 today](https://github.com/openebs/openebs/blob/master/k8s/dedicated/tutorial-ubuntu1604-vagrant.md) and let us know what you think!\n\nHangout with us and help us with your valuable feedback at [slack.openebs.io](http://slack.openebs.io)\n","notHasFeatureImage":false,"slug":"openebs-sprinting-ahead,-0.2-released"},{"id":136,"title":"Software Defined Storage — finally","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"11-03-2017","tags":["DevOps","Docker"],"excerpt":"In this blog, I’ll discuss what went wrong — why we didn’t achieve the promise of software-defined storage — and why software-defined storage is now, finally, possible.","content":"\nMany years ago there was a flowering of what we called software-defined infrastructure. Those of us at the forefront of the trend were no doubt encouraged by the success of Martin Casado and the Nicera team who quickly went from Ph.D. thesis to $1.26bn purchase by VMware thanks to the promise of software-defined *networking*. In hindsight we were misguided, but a handful of us built companies built on the premise — why not *storage*?\n\nToday we finally have the ingredients in and around the storage industry to achieve what we all were shooting for 5–7 years ago. I’m so encouraged by these enabling trends that I’ve gotten back into storage even as it has fallen out of investor favor. Cutting-edge users are now achieving the cost savings and boost to their agility we all promised — and truly believed — were around the corner years ago.\n\nIn this blog I’ll discuss what went wrong — why we didn’t achieve the promise of software-defined storage — and why software-defined storage is now, finally, possible.\n\nThere are at least a handful of reasons that storage failed to achieve the true promise of software-defined storage — that we did not live up to the radical flexibility of virtualized compute for example, where compute jobs were largely freed from their underlying hardware:\n\n1. The lack of portability of the controllers themselves\n2. The inherent stickiness of the underlying data\n3. Inability to deliver performance in a dynamic environment\n4. Immature common standards\n\n****Controller portability and data stickiness:****\n\nStorage controllers have to be able to deliver the policies you want — share these folders with these thousands of people for example — while keeping the underlying data safe irrespective of hardware failure, bit rot, user error, and more.\n\nFor a couple of reasons we in the industry at the time were unable to make our controllers truly software-like in the way that they were deployed in data centers: one, we didn’t sufficiently separate the policy engine or storage scheduler from the care and feeding of the underlying data on disk; two, in part because of this as well as the way that hardware drivers interact with operating systems, our storage controllers tended to need to be bare metal installs, and did not run well in a virtual machine. Perhaps more importantly, because our controllers needed to run on the same systems where the data was stored, they could only ever be as fluid and dynamic as the data itself. In other words, we were chained to our hardware in part by the physics of reading, transporting, and writing data.\n\n****Performance amidst dynamism:****\n\nInformation technology design is often a game of pass the bottleneck. For any given system there is one bottleneck or constraint that limits much of overall performance and hence that throttles the ability of the application to serve user needs. Historically the bottleneck has often been storage — and the rise of virtualization just tightened further this bottleneck thanks to the I/O scrambling that hypervisors perform on the read/ write patterns of the applications running on them.\n\nAnd yet if you are to treat storage — and storage controllers — as software that itself can be dynamically provisioned and moved about then you must be able to deliver performance from underlying systems irrespective of where the controller software is located. Perhaps more importantly, you need to be able to interpret the requirements of the applications themselves and deliver the combination of IOPS and latency they need to meet the requirements of their end-users.\n\nThis problem has aspects of the traveling salesman problem, which is to say, it is not entirely solvable. The way it has been solved in practice is that storage has remained bound to particular sets of typically hugely over provisioned hardware and the combination of applications and underlying storage has itself had to be controlled through affinity rules in the compute schedulers from VMware and OpenStack and others. In the container world, for the most part, we have dealt with the challenges of delivering storage performance to so-called stateful containers **by not having stateful containers**. The vast majority of containers that are deployed don’t actually rely upon data storage in the way that databases for example do. Those containers that are stateful typically tightly couple the underlying storage to containers, thereby removing much of the dynamism and ease of management that was a primary point in moving to containers in the first place.\n\n****Immature common standards****\n\nAs if the inherent difficulty of somehow addressing QoS for storage, while allowing controllers themselves to become more flexible, was not hard enough, the industry structure of IT 4–5 years ago itself made it more difficult to deliver software-defined storage. VMware tried hard to get everyone on the same page via their VASA APIs, which was a way to pass information about applications to the storage and for the storage to essentially sign-up for the performance needed, however, this effort ended up making less transparent and more proprietary the DNA, or operating instructions, of the software-defined data center. Perhaps because these operating instructions themselves were so opaque they never caught on in software-defined data centers.\n\nToday the Kubernetes community is probably our best shot for having a set of commonly accepted application definitions that flow into the infrastructure, to actually deliver software-defined infrastructure. In this case, the DNA is human readable YAML and is managed by a set of open source technologies.\n\nQuick note — pod and resource definitions are not fully fleshed out by Kubernetes for storage. So you can do some basic things, such as limiting the amount of storage by user or application or pod, however storage specific QoS is not yet supported in these definitions. This is a work in progress.\n\nNonetheless, what is possible today gives a good idea of what is coming. For example:\n\n![human readable code example](/images/blog/software-defined-storage-finally-example-code.png)\n\nAs you can see, very simple, human readable, and change controllable easily via GitHub or other systems, which is fundamental to achieving a high degree of automation and control.\n\n**The unevenly distributed future**\n\nWhile this blog and self-assessment of where we got to in software-defined storage may be a bit depressing, there are signs of hope.\n\nWithin the Kubernetes community, for example, hardy pioneers with deep technical expertise such as Pearson are using solutions like StackStorm and much else to build truly developer defined infrastructures that include the use of containers for stateful workloads.\n\nWhat I learned from Pearson and other StackStorm users led me to look for storage intellectual property — and the teams that built it — that could enable the storage freedom promised by software-defined Storage. Specifically, I went looking for technologies that could virtualize or containerize storage controllers while somehow ensuring the delivery of QoS.\n\nIn CloudByte I found a solution that today delivers fine grained control of QoS via an architecture that features virtualized storage controllers. What this means is that the controllers themselves can be live migrated, for example, while continuing to serve storage. This is how CloudByte delivers non-disruptive upgrades — and it also means that you can migrate pieces of your data center from on-premise to the cloud and back — with confidence. However, without QoS controls moving your controllers around would be madness — and so the deep understanding of QoS and the ability to set QoS SLAs by the user or by volume for example is crucially important.\n\n![Storage controller migrating from one site to another](/images/blog/storage-controller-migrating-from-one-site-to-another.png)\n\nIn the above image, I show a storage controller migrating from one site to another.\n\nAnd what CloudByte does for today’s primarily scale-up workloads — largely “pets” — our emerging open source project called OpenEBS will do for emerging scale-out workloads — so called cattle — through “containerized storage for containers.”\n\nIn short — as we plan to further illustrate through use case stories especially of hybrid cloud and backup use cases — in CloudByte / OpenEBS I’ve found not just the building blocks but the first instantiations of the future we all dreamed of several years ago.\n\nTime to reboot — and re-energize; storage now has the ingredients needed to unshackle users and enable them to achieve much more dynamic IT while keeping control of their data. Join me by taking a look at CloudByte and OpenEBS today. I look forward to your feedback.\n","notHasFeatureImage":true,"slug":"software-defined-storage-—-finally"},{"id":137,"title":"Using OpenEBS to build a true on-premise container service","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-02-2017","tags":["AWS EBS","Container as a Service","OpenEBS","Persistent Storage","Kubernetes"],"excerpt":"The top questions that could be lingering on the enterprise architect mind in an enterprise are","content":"\nThe top questions that could be lingering on the enterprise architect mind in an enterprise are\n\n- How do I build a true container service in my enterprise?\n- Do I really have to depend on the popular clouds such as Google, AWS, DigitalOcean to start my container journey?\n\nBuilding container services on AWS or GCP is practical today. These cloud platforms provide the infrastructure to start building your container platform. You get VMs for hosting K8s minions and masters and persistent disks (EBS disks or GP disks) to persist the block storage of your applications. If you want to build a similar container service in your enterprise on-premise data center, what are the choices for container infrastructure?\n\nWell, the troubling infrastructure piece to build the true container service in your enterprise is the persistent storage for your applications. No surprise there. VMs or bare metal can be deployed using well-known tools such as OpenStack and KVM, but for deploying EBS or GPD equivalent, you would need something like OpenEBS.\n\nOpenEBS helps the enterprises to build an AWS EBS equivalent or GPD equivalent platforms on-premise. OpenEBS is architected to ease the provisioning and management of persistent volumes at scale. It will have most of the elements of the popular AWS EBS. I had discussed the comparison between AWS EBS and OpenEBS with few community members in our last meetup. The slides are posted at [https://www.slideshare.net/OpenEBS/openebs-containerized-storage-for-containers-meetup-2](https://www.slideshare.net/OpenEBS/openebs-containerized-storage-for-containers-meetup-2)\n\nContainerization of block storage volumes gives the benefit of flexible storage upgrade schedules, treating the storage volume as part of your K8s POD, etc. However, there is more to consider as benefits of OpenEBS. OpenEBS is “Open Source” EBS. A quick feature comparison is as follows\n\n![Comparing AWS EBS & OpenEBS](https://cdn-images-1.medium.com/max/800/1*uu_mIhdqobjf3ftNOtf8KQ.png)\n\nNext, provisioning and consuming persistent volumes through OpenEBS is very similar and simple to that of AWS EBS\n\nIn AWS EBS a user creates and attaches a disk to an EC2 instance. The nuances of underlying storage protocols are hidden underneath and not exposed to the consumer/user.\n\n![Connectivity in AWS EBS](https://cdn-images-1.medium.com/max/800/1*zShnxODcXjTNu-X-qsJa5g.png)\n\nFlow of provisioning and consuming persistent block volumes on AWS EBS\nIn OpenEBS, it is very similar. Once the OpenEBS volumes are expressed as intent in the application YAML config file, the volumes are automatically created on the OpenEBS platform, mounted on K8s minions, and the persistent storage is made available to the application.\n\nExample of OpenEBS volumes getting consumed through iSCSI:\n\n![Connectivity in OpenEBS](https://cdn-images-1.medium.com/max/800/1*Mh9MzX5a_YbV9K_LR8EynA.png)\n\nAn example of OpenEBS volumes getting dynamically provisioned and consumed using k8s-openebs provider is shown below\n\n[https://github.com/openebs/openebs/blob/master/k8s-demo/my-nginx-pod-on-openebs.yaml](https://github.com/openebs/openebs/blob/master/k8s-demo/my-nginx-pod-on-openebs.yaml)\n\nThe next comparison of OpenEBS to AWS EBS is how the volume snapshots are managed for data protection.\n\n![Snapshots on AWS EBS vs OpenEBS](https://cdn-images-1.medium.com/max/800/1*elAnAeYarCwxeCEyXv_Xow.png)\n(***Comparing snapshots management on OpenEBS to that of AWS EBS***) \n\nAs shown above, using OpenEBS, you will have a standard S3 snapshot upload capability so that you can choose your S3 provider. With on-premise S3 technologies like Minio, you can have the container backup infrastructure also as on-prem.\n\nOn the advanced features comparison, OpenEBS steps up to match AWS EBS. Granular QoS control, advanced IOPs management features such as using burst and credit IOPS are some of the features that will be home at OpenEBS\n\n![Other features comparison of AWS EBS & OpenEBS](https://cdn-images-1.medium.com/max/800/1*1WGi8-GdTamykwgnJ0lqjw.png)\n\nBuilding reliable storage platforms in open source is a hard thing. With all the expertise in serving large enterprises, the team seems to be up for the challenge. We hope to see the increasing levels of community engagement in the months to come!\n","notHasFeatureImage":true,"slug":"using-openebs-to-build-a-true-on-premise-container-service"},{"id":138,"title":"Torus (from CoreOS) steps aside as Cloud Native Storage Platform. What now?","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"20-02-2017","tags":["DevOps","Docker","Storage Containers","Kubernetes"],"excerpt":"Torus (from CoreOS), that aimed at providing container-native distributed storage announced that it would discontinue development due to lack of traction.","content":"\n[Torus (from CoreOS)](https://github.com/coreos/torus), which aimed at providing container-native distributed storage announced that it would discontinue development due to lack of traction. It just goes to show the dynamics of the open source projects and how business needs could influence them.\n\nTorus attempted to solve one of the core infrastructure problems, and like any projects that aim to solve the infrastructure problems — compute, network or storage, are slightly harder problems to crack with only a handful of people really venturing into solving them. Maybe, this inertia is also due to the fact that, when infrastructure fails, hell breaks loose — we are all too familiar with those network outages and data loss situations where Database teams spend days recovering data.\n\nHarder problems require renewed focus, at least in the initial stages to get the product off the ground for the basic use case. The entry barrier must be minimal, if not totally absent. Once the core value is established, in terms of ease of use or scale and just solving a problem in an intuitive way, it graduates into the adoption phase.\n\nWith containers changing the landscape of applications, they pose an additional set of requirements on the storage platform, which are covered in my earlier post — [Emerging Storage Trends for Containers](/blog/emerging-storage-trends-for-containers). *Any container-native storage platform developed today should be distributed and natively integrated into container orchestration platforms like Kubernetes*. Torus got these aspects right.\n\nWhile suspending further Torus development, I find it interesting that the committers at CoreOS mentioned Rook.\n\n[Rook](https://github.com/rook/rook), like [GlusterFS-container](https://github.com/gluster/gluster-containers), are not container-native, they provide wrapper projects that make it easy to deploy alongside Kubernetes. In the short-term towards the journey of taking containers into production, this retro-fitting approach may be OK! If there is enough traction to refactor the existing projects (Ceph) as we move forward.\n\nBut as we saw with compute virtualization paving the way for software-defined storage, there is a need for container defined storage. PortWorx and OpenEBS alone currently come close to being container-native storage platforms, built ground-up using containers by teams that have built enterprise and cloud storage platforms.\n\nOpenEBS is a distributed block storage that supports both hyper-converged and dedicated deployment models. However, the implementation of the block storage with OpenEBS follows a different approach for block replication (forked out of [Rancher Longhorn](https://github.com/rancher/longhorn)), which optimizes the meta-data processing of the distributed blocks.\n\nOpenEBS also inherits the data protection capabilities, pushing backups onto Amazon S3. We are currently working on EBS like API for provisioning the storage from K8s. With its cloud integration capabilities, we envision the capabilities supported by OpenEBS to enable our customers to build On-prem container environments that are compatible to be moved into the Cloud and vice-versa.\n\nOpenEBS Storage is delivered through VSMs which are essentially containers — frontend and backend, that can be scheduled by Kubernetes along with the Application Pods. The storage specific parameters are fed into the scheduling algorithms through our OpenEBS storage orchestration layer — maya. This container mode of deployment also reduces the burden of managing a separate storage network. The building blocks of the OpenEBS were covered in this post — [OpenEBS building blocks](/blog/openebs-building-blocks-rancher-longhorn)\n\nIf you were planning to contribute to Torus or just interested in getting your hands dirty with container-native storage platform, please take a closer look at [OpenEBS](https://github.com/openebs/openebs), which is completely open source.\n","notHasFeatureImage":true,"slug":"torus-(from-coreos)-steps-aside-as-cloud-native-storage-platform.-what-now?"},{"id":139,"title":"Deployment modes of OpenEBS","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"13-02-2017","tags":["Container","Docker","Kubernetes","OpenEBS","Storage For Containers"],"excerpt":"OpenEBS supports two modes — Hyper-converged and dedicated. The deployment mode really depends on where and how you want to use OpenEBS.","content":"\nOpenEBS supports two modes — Hyper-converged and dedicated. The deployment mode really depends on where and how you want to use OpenEBS. If you are adding block storage capability to existing Kubernetes minions, hyper-converged mode is most desired, so that you can use the existing hardware as is. If the desire is to get a full fledged EBS type functionality to your on-premise cloud or container needs, then dedicated storage servers for OpenEBS is a better choice.\n\nIn the hyper-converged mode, OpenEBS Maya hooks into the K8s master and minions, hooking into scheduling algorithms for creating OpenEBS VSMs. When used in dedicated mode, the provisioning API are exposed via the OpenEBS Maya master. Dynamic Provisioning of the storage can be enabled using the volume plugin drivers or use the EBS volume plugin as in the case with K8s.\n\n![OpenEBS — Hyper-converged mode](https://cdn-images-1.medium.com/max/800/1*MxM5MmWCB_5mmy7A5bor6Q.png)_(OpenEBS — Hyper-converged mode)_\n\n![OpenEBS — Dedicated mode](https://cdn-images-1.medium.com/max/800/1*MAbRf5rJfv8w_OvZz02q7g.png)_(OpenEBS — Dedicated mode)_\n\nIn both modes, Flannel plays an important role in OpenEBS deployment for storage networking. A lot of issues are yet to be discovered in this area.. yes, this is just a start.\n","notHasFeatureImage":true,"slug":"deployment-modes-of-openebs"},{"id":140,"title":"Containerization meetup - Containers for storage too","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"15-01-2017","tags":["Containerized Storage","Docker","Kubernetes","OpenEBS","Storage Containerization"],"excerpt":"I had the opportunity to talk to a very good group of technologists, DevOPs users in the Digital Ocean Containerization Meetup . Docker 1.13 details and Kubernetes deployment tips were fantastics.","content":"\nI had the opportunity to talk to a very good group of technologists, DevOPs users in the [Digital Ocean Containerization Meetup](https://www.meetup.com/DigitalOceanBangalore/events/236353004/) . Docker 1.13 details and Kubernetes deployment tips were fantastics.\n\nI talked about the containerization for storage. The presentation slides are published [here](http://www.slideshare.net/UmasankarMukkara/openebs-containerized-storage-for-containers). Slide #6 talks about the advantages when the storage volumes are containerized, just like the advantages when the applications are containerized. In the storage containerization, the storage software becomes a micro service. Storage protocol (iSCSI in this case), replication, QoS, Encryption are micro services that are better managed and served when containerized. This form of containerization leads to a true non-disruptive storage upgrade possibility in production.\n\nImagine, there are thousands of storage volumes in production serving thousands of containerized applications. If a storage upgrade has to happen that actually is needed only few volumes, such as a special snapshot or encryption feature or bug fix, then only those containers need to be upgraded. The maintenance window scheduling will be the simplest in such scenarios. To make this possible, OpenEBS abstracts the storage functionality into the user space (slide #7) and each volume is dedicated with a separate storage process in the form of containers… Micro services architecture to the storage, delivered.\n\nI also talked about the building blocks of OpenEBS.\n\nOpenEBS uses or builds on:\n\n- Docker — for achieving containerization. [Jiva](https://hub.docker.com/r/openebs/jiva/) is the docker image\n- Longhorn Rancher — For basic storage replication and distributed scale-out needs. QoS, encryption comes from here.\n- Nomad — Clustering capability among storage hosts as well as storage pods or VSMs is served by Nomad\n- Consul — for the cluster db\n- Flannel — for the container networking needs. We use the flannel’s intelligence for picking up the IP addresses for VSMs automatically, VLANS and other networking stuff. The database portion of flannel is not needed as OpenEBS has the centralized config db for Maya (consul implementation)\n\n![OpenEBS Building Blocks and Integration](/images/blog/containerization-meetup-building-blocks.png)\nAs far as the integration points of OpenEBS are concerned, the provisioning in integrated into k8s. The provisioning of the OpenEBS can be done through the [K8s iSCSI volume interface](https://kubernetes.io/docs/user-guide/volumes/#iscsi) or through the [K8s AWS EBS interface](https://kubernetes.io/docs/user-guide/volumes/#awselasticblockstore). Yes, OpenEBS exposes AWS EBS API. Any orchestration layer or application that knows to connect to and use AWS EBS, will work with OpenEBS.\n\nAs we progress from this early stage, we look forward to work closely with the Docker and k8s user communities.\n","notHasFeatureImage":true,"slug":"containerization-meetup---containers-for-storage-too"},{"id":141,"title":"Emerging Storage Trends for Containers","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"11-01-2017","tags":["DevOps","Docker","OpenEBS","Startup","Storage"],"excerpt":"The smiling docker whale is everywhere these days. You either have ridden on it or you want to know how to ride on it. As far as operations teams are concerned the docker whale is just teasing them.","content":"\nThe smiling docker whale is everywhere these days. You either have ridden on it or you want to know how to ride on it. As far as operations teams are concerned the docker whale is just teasing them. In the recent meetups and conferences I attended here, very few hands rise when asked if they have deployed containers in production.\n\nWell, it has taken over a decade and half at least for the shift from servers to virtual machines. And even now, I see a lot of caution in the enterprises to move completely into the public cloud. Though the technology around virtualization is well established, the real return$$ of moving onto public cloud aren’t as great as initially expected.\n\nA similar transitional trend, but at a much faster pace is being seen around the containers. Docker has done a phenomenal job in breaking the entry barrier for playing with containers, by addressing the Container Eco-System components for Developers to quickly build, test and package the applications via containers.\n\n![Interest over time](https://cdn-images-1.medium.com/max/800/1*c8cxwXMmU93xXAiK4Rs0BA.png)\n\nWhile it is very easy to setup and run containers, are dockers really ready for the production work load? The following diagram pretty much sums it up.\n\n(Credit: [http://bicarait.com/2016/11/01/running-your-docker-using-vsphere-integrated-container/](http://bicarait.com/2016/11/01/running-your-docker-using-vsphere-integrated-container/))\n\n![Containers in development & production](https://cdn-images-1.medium.com/max/800/0*nDL6ATRys2vPH8m9.png)\n\nLearning from the recent past, the cloud was led by the compute virtualization, but it wasn’t a true cloud until the network and storage also were fully virtualized, giving way to software-defined Storage and software-defined Networking. These days it is about SDDC, which involves significant improvement in the orchestration of infrastructure and the services running on them. For those of us that have contributed and implemented the cloud, and operated them in production — the journey was exciting and in some instances nail biting!!\n\nI keep hearing developers say, I don’t care about the infrastructure. That’s great, but it just means that someone else is taking care of it, usually the operations team! I wonder if the DevOps is bringing developers closer to operations or Operations closer to Developers? I bet it is the Operations Team getting closer to Developers and making the life of Developers much easier.\n\nThe Developers are excited about the stateless containers or server-less architectures or lambda services. All of these are great architectural patterns, but all these eventually depend in some form on some stateful containers or servers that are running over robust infrastructure with extremely low latency.\n\nThe Operations Teams are the one that need to define this robust infrastructure for running the containers. Unlike VMs that were provisioned (or self provisioned on designated infrastructure) for specific use-cases, the container infrastructure must come with a brain of its own, where the containers are provisioned and moved depending on the load and performance parameters.\n\nAn application or service in the container world, will be a set of interconnected containers (stateful or stateless — much like a K8s Pod), that will require compute, network and storage. And for these Application Pods to be portable, the infrastructure also needs to move along with them. It is time for programmable infrastructure to go mainstream. Just as virtualization paved way for software-defined networks and software-defined storage, we are now seeing the need for container defined network and container defined storage.\n\nFor a recent [Bangalore Docker meetup](http://neependra.net/?p=2141), I took a shot at looking through the various storage startups that are trying to build storage for the containers. The slides are available here : [Emerging Storage Trends for Containers](http://www.slideshare.net/kiranmova/emerging-storagetrendsforcontainers)\n\nThe storage trends can be summarized as follows:\n\n(1) (Slide#22) **Elastic Storage Infrastructure** — The storage can be horizontally scaled, much like the docker hosts in docker swarm. The technology to implement the Elastic Storage is already in production with many vendors supplying **_Software-Defined Scale-out/Distributed Storage_** in the cloud environments.\n\n(2) (Slide #23) **Ease of Accessing the Storage**— The volume of volumes required by the containerized applications will be manyfold compared to volumes used in the VM storage. And the volumes will be more portable. Operations team should be shielded from having to mount/unmount volumes or datastores. **_The integration into the orchestration layers for mounting the storage and providing access to the volumes, without requiring additional software changes will be paramount._** The TCMU/iSCSI, NBD/NFS interfaces are two different approaches with each coming with its own nuances w.r.t isolation vs ease.\n\n(3) (Slide #24) **Hyper Converged Storage** — For storage to be hardened, the developer setups need to be and **_avoid managing silos of infrastructure for compute, network, and storage._** The concept of lightweight storage software will emerge which will mount distributed external storage or cloud storage onto the local machines (with caching).\n\n(4) (Slide #25) **Hybrid Clouds / Storage** — Put the storage in the right place and move it around depending on the demands of the application and economics. A first in this place would be to move the snapshots into S3 when the persistent disks are used for stateful containers. The industry has established that to save money, a mix of clouds is going to be used with the apps moving into different container environments. The storage platform should be able to run alongside different clouds or have the ability to inter-operate. **_Seamless Storage Migration within and across clouds is a must._**\n\n(5) (Slide #26) **Containerized Storage** — One of the best innovations is the way the containers are defined and deployed. The storage software also will reap these benefits by containerizing the storage. **_Version of the storage being used will become another parameter to be defined in the intent specs along with capacity and performance (QoS)._** Containerization also helps with isolation and ease of upgrades in shared environments.\n\n(6) (Slide #27) **Keep up with the core storage innovations** — Flash is already mainstream, with CIOs questioning if they still really need the SAS Tiers. With NVMe, cache is becoming more accessible both in terms of $$ and the performance boost it can provide to the Storage Software. **_The low latency demands from the containerized applications will be guaranteed via the use of NVMe Flash for read/write caching of data that is possibly stored on disks or remote cloud storage._**\n\nIt is euphoric to see the new developments being made in the opensource for the storage, networking, and orchestration layers, apart from the container runtime itself!\n\n2017 may well be year where we will start seeing containers in production as the infrastructure pieces for Containers mature.\n","notHasFeatureImage":true,"slug":"emerging-storage-trends-for-containers"},{"id":142,"title":"OpenEBS Building blocks — Rancher Longhorn","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"06-01-2017","tags":["Container","DevOps","Longhorn","OpenEBS","Docker"],"excerpt":"In the previous blog post, I shared the initial details of OpenEBS. In this post, I will continue to discuss the OpenEBS technology building blocks and touch upon the details of Rancher longhorn and why we chose longhorn.","content":"\nIn the [previous](/blog/openebs-the-containerized-storage) blog post, I shared the initial details of OpenEBS. In this post, I will continue to discuss the OpenEBS technology building blocks and touch upon the details of Rancher longhorn and why we chose [longhorn](https://github.com/rancher/longhorn).\n\nOpenEBS platform contains three core building blocks:\n\n- An orchestration platform, Maya, that works with Kubernetes and manages thousands of volumes with ease.\n- Containerized storage volumes called Virtual Storage Machines or VSMs and\n- Maya managed backing stores or data stores residing either locally on OpenEBS hosts or remotely over network\n\n**_Just to recap, why storage containerization?_** With storage containerization, the storage upgrades are flexible, easy, and effective. The containerization of storage means that the core functionality of storage (like front end protocol ISCSI, snapshotting, replication, backup) is abstracted into a Docker container and managed outside the kernel. A software patch to correct or enhance the replication behavior of a volume does not affect the other volume in the same host. Each of these Docker containers do a specific job of either running [gotgt ](https://github.com/gostor/gotgt)iSCSI or running a [longhorn ](https://github.com/rancher/longhorn)replica. The storage software is built as [Docker image](https://hub.docker.com/r/openebs/jiva/) and is the core/essence of OpenEBS technology. Hence, we named it “Jiva” (meaning “life”, [Wikipedia](https://en.wikipedia.org/wiki/Jiva)).\n\n**_About VSM:_** Virtual Storage Machine is the logical set of storage pods that encapsulates the entire functionality of the life cycle of a volume. The components of an OpenEBS VSM is shown in the below picture.\n\n![Fig: OpenEBS VSM components](https://cdn-images-1.medium.com/max/800/1*-Bl0JyjyNdVe_bp6YI-n6w.png)\n\nA VSM contains as many storage pods as the number of data copies of the volume. Each storage pod has at least one container for replica and optionally has a container for exposing the storage access protocol (iSCSI, NBD etc). We are using a fork of rancher/longhorn software to manage the replication among the storage pods and a fork of gostor/gotgt software to provide iSCSI interface.\n\n## What is longhorn ?\n\nLonghorn is a simplified block storage software, implemented in golang, that stores the entire volume as a single linux sparse file. The sparse files provide thin provisioning behavior. Formatting with QCow2 adds the CoW feature to the data. It is lean and provides an AWS EBS style snapshot functionality. Longhorn has two subcomponents, longhorn controller (LHC) and longhorn replica (LHR). LHC takes care of storage connectivity, replication, rebuild, encryption, etc while LHR does snapshotting, backup, QoS, etc.\n\nLHC and LHR can be deployed in two modes.\n\n- _Hyper-converged container model,_ where LHC and LHR are on the same host as that of compute or Docker Host. TCMU is used for block storage volume drive emulation on Docker Host.\n- _Remote storage model,_ where LHC and LHR are on separate storage host. The compute Docker Host connects to LHC using iSCSI. iSCSI client is used for block storage volume drive emulation on Docker Host.\n\nThese two models are shown below\n\n#### LHC and LHR are on the Docker Host\n\n![Fig. Longhorn deployment mode : Hyper-converged](https://cdn-images-1.medium.com/max/800/1*nlswAfJqgqaWRJpKYLr_jA.png)\n\n_Note: Minimum linux kernel version required for hyper-converged mode is 4.4_\n\n## LHC and LHR are on the remote Storage\n\n![Fig: Longhorn deployment mode : Network storage](https://cdn-images-1.medium.com/max/800/1*wB_PG-Y_jZm8lMmSzKJAww.png)\n\nA third mode is also possible, where LHC runs on the Docker Host and LHR runs on the remote storage host, the discussion about this is for a later day.\n\nLonghorn replica uses 4K as the underlying block size and is a chain of differencing disks among the live data and snapshot data. The backup is done in a AWS-EBS style, where only the changed blocks are copied to the remote location (like S3) using 2M block size.\n\n## Why we chose longhorn for OpenEBS?\n\nWe wanted to implement a simple block storage engine, in user space, that can be containerized. Rancher had spent quite an amount of effort in just doing that. It is written in golang too. We found it to be thin, working, and fit. We integrated gotgt and longhorn for the basic use case of OpenEBS. We are thrilled to find great support from longhorn team in this journey. Thank you, Rancher. Though [openebs longhorn](https://github.com/openebs/longhorn) is forked at the moment from Rancher longhorn, we intend to push back the changes to the mainstream longhorn and contribute there. We plan to add the functionality of flash caching, S3 integration, RDMA support, cache tier-ing to remote storage, etc to longhorn in the days and months to come.\n\nThe next blog post will discuss the deployment modes of OpenEBS with containerized longhorn or jiva.\n","notHasFeatureImage":true,"slug":"openebs-building-blocks-—-rancher-longhorn"},{"id":143,"title":"I’ve become a Chairman — of CloudByte / OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"04-01-2017","tags":["DevOps","Docker"],"excerpt":"I’m excited to announce that I recently joined CloudByte, a developer of storage solutions for enterprises and service providers and the originator of the new OpenEBS project.","content":"\nI’m excited to announce that I recently joined CloudByte, a developer of storage solutions for enterprises and service providers and the originator of the new OpenEBS project. While my role is Chairman of the Board of Directors, I’ll be rolling up my sleeves and doing whatever I can to help.\n\nYou might ask — why? And so what?\n\n## 1. **YAML = truth**\n\nWhen I helped popularize the notion of software **defined** open storage years ago as founding CEO of Nexenta — one of the concerns was “who sets the definition?” At the time VMware was pushing themselves for that role. It was a bottoms up — infrastructure up — approach that never fully caught hold.\n\nIt was a pre DevOps approach of doing things that often ended in the old familiar:\n\nThese days the intent of the developer and the requirements of the application are being expressed via flavors of YAML that Kubernetes, Docker Swarm, Mesos, StackStorm, Rancher and others leverage.\n\nCloudByte, via the OpenEBS community that builds on many of the technologies that CloudByte has been building and selling for years, is well positioned to be the preferred solution for containerized storage for containers not just because of the underlying QoS isolation bits — though that heritage is important — but also because we and the OpenEBS community are 100% focused on manifesting the intent and latent requirements of the application.\n\nTake a look at OpenEBS here: [www.OpenEBS.io](http://www.openebs.io/). It is early however already gathering some great contributors and users. Please let us know what you think.\n\n## 2. **I like money**\n\nNo, it isn’t that I’m joining the board for the money. News flash — early stage start-ups don’t have money lying around to pay board members, actually the cash flows the other way :) Rather, CloudByte’s technologies have been proven by demanding enterprises and service providers.\n\nThis existing customer set — and their increasing commitments as confirmed by a far and away record quarter of sales last quarter — gives us a foundation to build upon. We intend to extend our success in the scale up storage space via our flexible business model and superior technology — especially for use cases where our per volume QoS management matters to users.\n\n## 3. **Team time!**\n\nMost importantly, as I’ve come to know the CloudByte team I’ve formed a commitment to them and their vision for the kind of company we can build. I feel honored to support our CEO and co-founder Uma — he’s brilliant, he is stubborn, and he is relentlessly making real a vision for what storage can be — as an **enabler** of the shift towards cloud native and DevOps as opposed to an **inhibitor**. It didn’t hurt that the new board hit it off immediately and that Raj from Fidelity / EightRoads and Sandeep from Nexus have great entrepreneur-friendly reputations as well as massive funds supporting them.\n\nThere will be much more to come as we share our thoughts about strategy and, maybe more importantly, as [OpenEBS](http://www.openebs.io/) community members lay out what they’d like to see in containerized storage.\n\nUntil then — please get in touch. Hang out with us on [the community](http://www.openebs.io/) or just ping me via twitter @epowell101 or otherwise. Let’s work together to make storage — finally — something that does what it is supposed to do despite all the challenges inherent to providing persistence to today’s ever more dynamic environments.\n","notHasFeatureImage":false,"slug":"i’ve-become-a-chairman-—-of-cloudbyte-/-openebs"},{"id":144,"title":"OpenEBS — The containerized storage","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"01-01-2017","tags":["DevOps","Docker","Kubernetes","Rancher","Golang","OpenEBS"],"excerpt":"In the infrastructure space, the compute environment is always the first to lead the change. Docker has brought in the new thinking into every DevOPs administrator and application developer.","content":"\nIn the infrastructure space, the compute environment is always the first to lead the change. Docker has brought in the new thinking into every DevOps administrator and application developer. CIOs across the spectrum are beginning to include Docker into their policies. As the IT environment begins to adopt containerization into mainstream, there are holes that still need to be plugged in, specifically in the storage space.\n\nI am excited to write the first details on OpenEBS ([www.openebs.io](http://www.openebs.io/)) project.\n\nAfter 4 amazing years of building commercial storage products, I recently took a giant leap into the open source model of building infrastructure software. We recently launched OpenEBS project with the goal of building the developer friendly storage using coolest available infrastructure pieces underneath .. be it Docker, k8s, Rancher LongHorn, Nomad, Terraform, etc.\n\n## So, what is OpenEBS, in short ?\n\nOpenEBS offers persistent block storage with the following features:\n\n- Containerized block storage using Docker containers. We call them VSMs or Virtual Storage Machines, a concept similar to k8s PODs.\n- A highly scalable storage orchestration platform that spins the storage volumes seamlessly and manages them effortlessly\n- A simple, yet high performing distributed block storage designed with best caching via the NVMe optimizations\n\n## Why containerized storage?\n\nSimple answer is that even the storage volumes have software associated with them for their regular functions and this software needs to be managed at volume level.\n\n![Monolithic vs Containerized Storage](https://cdn-images-1.medium.com/max/800/1*OoQnpEsGf_ovb5BFnGI8hA.jpeg)\n\nWhen software upgrades happen at storage host level, all volumes' behavior will change simultaneously, which may not be the desired result. Similarly, upgrade maintenance windows may not be the same for all applications or storage volumes. We often observe that it is very difficult to get a convenient window that satisfies all storage volumes or associated applications.\n\nWith containerized storage, storage upgrades becomes simple and easy, just like application upgrades with Docker containers.\n\n## Maya, our new storage orchestration platform\n\nKubernetes, Docker swarm, Rancher cattle, Nomad and other orchestration platforms do a good job of managing the lifecycle of compute containers and initial provisioning of network and storage. However, the storage infrastructure management, when scaled is a big beast in itself. Storage volumes need to be persistent to the application but they need to be volatile in the backend. Storage volumes need to be scheduled on various hosts based on the capacity and IOPS availability and these volumes may need to be moved on the fly as the usage goes up.\n\n_Maya in Sanskrit language means “\\***\\*Magic\\*\\***”. Maya will seamlessly integrate storage management functionality into existing container orchestration layers for provisioning, scheduling, reporting, rolling upgrades etc., and provide storage specific capabilities like data protection capabilities, migrating storage etc.,_\n\n## The building blocks of high performing, distributed block storage:\n\n**_Rancher longhorn:_** We chose to adopt and enhance Rancher longhorn as the basic building block of storage block intelligence in OpenEBS. I will write a separate blog about what is longhorn, it’s features and why we chose longhorn, but in short, longhorn employs a clever and simple approach to container data connectivity, data availability (replication), data protection (snapshot). And longhorn is written in GoLang.\n\n**_Gostor gotgt:_** One of the initial front ends for OpenEBS is of course the iSCSI. We chose gostor/gotgt as a good starting point. OpenEBS plans to add many new capabilities to gotgt like clustering support, performance optimizations etc.\n\n**_Bulk Caching layer through NVMe:_** The caching layer that we see in traditional storage systems is usually small in size. The recent advancements in flash technology made it possible to offer large capacities of flash at affordable prices. Now cost is not a deterrent to have terabytes of low latency flash storage. OpenEBS provides an intelligent caching technology which keeps the hot data in the large NVMe flash layer. Intel’s 3d XPoint is a good fit for this technology.\n\n## Community:\n\nCommunity is paramount. We hope to embrace a lot of friends, advisers, experts in this journey and successfully deliver the OpenEBS promise. Drop by at our [gitter channel](https://gitter.im/openebs/Lobby) and say Hi !\n","notHasFeatureImage":true,"slug":"openebs-—-the-containerized-storage"},{"id":145,"title":"Running through the mud looking skywards","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"2017-02-11","tags":["Education","Industry","Storage","OpenEBS"],"excerpt":"I always believe, bridging the minds of academia & industry has to go beyond the nuances of theory vs. practical.","content":"\nI knew we would be running through the mud looking skywards for those brightest stars. We were not at all disappointed with our first attempt at _Kuppam Engineering College_, India.\n\nI always believe, bridging the minds of academia & industry has to go beyond the nuances of theory vs. practical. We call ourselves the industry’s core and form the industry’s grey cells. What we need to do is make our problems open to the public. These problems becomes the new practice that needs to be picked up by our _alma mater_. Once selected, it will take no time for these problems to go viral among the students of these institutions. Let these young minds chart their own course; love, enjoy & live every moment when they traverse through this meandering amazon of practical problems. This becomes the fodder for the future theory allowing the current one to rest in peace.\n\nAbove sounds like a grand planning that resembles more like a trickle down effect. Well, the answer is a big ‘No’. Team at [**OpenEBS**](http://openebs.io/) believes there are many faster ways to achieve above in a time bound manner. Hence the formation of various internship programs where [**OpenEBS**](http://openebs.io/) approaches the Universities to get the latter on same plane that the former is passionate about.\n\n**At Kuppam, we could feel the freshness in the air, the disruptions that yearn to come out of the closet. We are all geared to lighten up this brilliance that is currently lying dishevelled.**\n\nOn a closing note, I challenge the myriad number of B.E projects & Ph.D. programs that are purposeless versus the likes of problems that the world wants us to solve. Join me in my effort to solve the problems that matter.\n","notHasFeatureImage":true,"slug":"running-through-the-mud-looking-skywards"}] \ No newline at end of file +[{"id":1,"title":"Mayastor 1.0 has arrived","author":"Prasoon Pushkar","author_info":"Prasoon works as a Product Marketing Manager in DataCore Software. His day to day activities revolve around ideating & executing positioning & messaging to grow OpenEBS & Mayastor. His free time is all about travelling, munching & playing FIFA","date":"18-01-2022","tags":["Mayastor","Kubernetes","Solutions"],"excerpt":"In this blog, we will go through Mayastor architecture, evolution of project over the years, benefits for the community and different use cases","content":"\nMayastor was started in late 2019 as a sub-project of OpenEBS and has been under development by the [MayaData](https://mayadata.io/) team as an advancement of earlier storage engines. Over the years, Mayastor remained in beta and the culmination of efforts of the engineering team of MayaData, the Cloud Native Computing Foundation and the vibrant open-source community led us to build the foundation of various features and stability. In that spirit, today we are proud to release the first community version of Mayastor . \n\nIn this blog, we will go through Mayastor architecture, evolution of project over the years, benefits for the community and different use cases.\n\n## What is Mayastor \n\nAs a sub-project of the [OpenEBS](https://openebs.io/) [CAS](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/) (Container Attached Storage) solution, the Mayastor storage engine is designed to provide persistent storage that is easy to manage and readily deployable. Mayastor is a cloud-native storage platform by MayaData that abstracts storage resources to enable persistent storage with low abstraction overhead for stateful Kubernetes applications. \n\nBuilt on the Container Attached Storage framework, the storage engine leverages a hyper-converged deployment model, where data is stored locally then shared with other nodes via synchronous replication.\n\n## Mayastor Architecture \n\nThe platform is built for cloud-native storage orchestration and is workload-driven to simplify Kubernetes storage for enterprises. Through a declarative data plane, Mayastor abstracts storage resources to enable persistent storage for Kubernetes applications. To do so, Mayastor storage engine components are deployed as containers in Kubernetes, allowing for simple scaling, provisioning and management of storage for clusters.\n\n**Control Plane -** The Mayastor control plane is primarily a single instance Kubernetes controller that implements both the Container Storage Interface (CSI) specification and private interfaces for the storage system. The controller is containerized and runs as deployment in Kubernetes. The control plane also includes CSI plugins deployed on each node to implement CSI protocol services. \n\n**Data Plane -** The Mayastor data plane creates a Nexus for every Persistent Volume Claim (PVC) loosely coupled to the storage class. Each Nexus instance acts as a virtual storage router, connecting the PVC with a Persistent Volume (PV). The Nexus instance provides an abstraction of the input-output controller while managing all requests for the Persistent Volume attached to the PVC. \n\n**NVMe and NVMe-oF-** Mayastor is designed to leverage the benefits of both NVMe and NVMe-oF to support low latency workloads for converged and segregated storage. With Mayastor, organizations can benefit from the performance capabilities of new generation high performance SSDs.\n\n**[Storage Performance Development Kit (SPDK)](https://spdk.io/) :** Developed and maintained by Intel, SPDK offers valuable libraries, tools and user-level I/O execution to enhance the performance of storage media in a data centre. Mayastor integrates with the SPDK, helping storage administrators leverage its features for improved throughput performance and efficient SSD storage. The platform also helps provision high availability storage using poll mode drivers and user-level I/O execution capabilities. \n\n## Why Choose Mayastor ? \n\nThe Mayastor data plane leverages the full performance potential of storage systems, delivering high IOPS with less than 10% overhead. The declarative data plane utilizes developments of NVMe and SPDK to enable easy storage management. The following section outlines various features and benefits Mayastor offers. \n\n## Benefits of Mayastor 1.0 \n\n**Reduced Latency – Meet IO-intensive Applications Requirements**\n\nMayastor is developed using an architecture that is targeted at meeting the storage performance requirements of IO-intensive Kubernetes applications. Mayastor uses a CSI driver with application and platform awareness to speed up migration for single and replicated volumes. \n\n**Guaranteed Efficient Memory Transfer** \n\nThe Mayastor data plane is implemented in Rust, which includes a safe compiler that guarantees efficient memory transfer. NVMe architecture also ensures that each SSD controller communicates with one ring per CPU. This ensures the I/O controllers aren’t prone to internal locking, enabling simpler memory management. \n\n**Cloud-Agnostic Storage – Eliminate Vendor Lock-in** \n\nMayastor storage controllers are orchestrated by Kubernetes, and data is accessed via containers. As the storage engine is deployed using a CSI driver, volumes can be provisioned using any container orchestrator or runtime, eliminating vendor lock-in. This construct allows organizations to use Mayastor to orchestrate any open-source storage service running on different Kubernetes distributions. \n\n**Flexible Horizontal Scalability** \n\nMayastor integrates with OpenEBS Logical Volume Management (LVM), which pools underlying volumes and divides them into immutable logical units. These units can be replicated and distributed across multiple availability zones, enabling flexible horizontal scalability for storage workloads. \n\n**Seamless storage replication** \n\nThe Mayadata storage controller simplifies the provisioning and replication of storage instances across pods, thereby enabling efficient deployment of cross-cloud storage for stateful applications. \n \n\n**Highly Available persistent data storage** \n\nMayastor relies on Intel’s industry-leading Storage Performance Development Kit to deliver reliable access to persistent storage services and devices. The storage engine offers resilient volume services through synchronous replication that guarantees availability. \n \n\n**Easy deployment and management of storage services** \n\nThe gRPC API simplifies operations management while providing security and easy configuration of the Mayastor data plane. The control plane implements application-aware data placement for fine-grained control over timeouts, restarts and errors of Kubernetes storage volumes. \n \n\n**Low-overhead storage abstraction** \n\nMayastor follows the Container-Attached Storage (CAS) pattern, which orchestrates storage as a workload in Kubernetes. With CAS, storage scales predictably, allowing workloads to operate seamlessly with the storage system. Besides this, as Mayastor is built from the ground up to leverage the benefits of NVMe storage, the platform further reduces performance overhead. \n\n## Mayastor Evolution \n\nThe section below explores Mayastor’s release history, noting the major milestones in the storage engine’s development. \n\n**V 0.1.0** \n\nThis Alpha release was intended mostly for testing and development purposes and featured persistent volumes that could be exported over ISCSI transport. \n\n**V 0.2.0** \n\nThe first pre-release version of Mayastor included various useful features, such as: \n\n- NVMe-oF Support - The storage engine had the capability to create and share a Nexus with an established front-end target. \n\n- Mayastor Node CRD - With the resource definition enabled, storage admins can configure observability and persistence for Mayastor nodes. \n\n- Rebuild Process - This mechanism allowed to synchronously bring in new replicas without disrupting IO workloads by scheduling concurrent rebuild tasks. \n \n\n**V 0.3.0** \n\nReleased on 12th August 2020, the second pre-release version of the storage engine included features such as: \n\n- NVMe-oF support for CSI plugins \n\n- Block mode for raw block access \n\n- Direct device access for Mayastor pools \n\n- Automatic replica replacement \n\n**V 0.4.0** \n\nReleased on September 4, 2020, the fourth major version included numerous enhancements and new features. Major enhancements included: \n\n- Restart tolerance for the rebuild process \n\n- Nexus I/O internal retries for failed operations \n\n- Automatic generation of Mayastor pool names \n \n\nNew features included: \n\n- Mayastor support for Kubernetes clusters built on K3s \n\n- Enhanced user documentation with simple navigation, improved UI and search \n \n\n**V 0.5.0** \n\nFeatures and enhancements of Mayastor 0.5.0 include: \n\n- Block device discovery through the gRPC API \n\n- Storage pool finalizers to prevent inadvertent deletion of volumes that contain replicas \n \n\nMayastor V0.6.0 and 0.7.0 were maintenance releases, while V0.8.0 was a patch release that included various stability fixes in preparation for the first major production release.\n\n\n**Mayastor 1.0** \n\nMayastor 1.0 is released now with multiple enhancements and stable features that make the storage engine respond more resilient to failures. Some features to be included in the release are: \n\n- TCP access support \n\n- N-way synchronous replication for workload protection \n\n- Support for Prometheus metrics and visualization in Grafana \n\n- CSI driver for volume management \n\n\n## Mayastor Use-Cases \n\n**Storage unification** \n\nThe Mayastor data plane can be deployed on any cloud running a Kubernetes environment. Storage teams can deploy storage controllers on multiple environments, on-premises and from cloud storage providers, and manage them from a central interface. \n \n\n**Low latency workloads** \n\nNVMe and NVMe-oF architectures reduce storage network latency and support high-speed storage media by eliminating bottlenecks caused by SATA connections. \n \n\n**Programmatic storage access** \n\nWith CAS, the storage controller does not have to make system calls for IOPS operations. Storage access is simplified since the controllers write to file and block storage devices directly from within the application. \n \n\n**Server consolidation** \n\nMayastor can manage heterogeneous workloads by orchestrating multiple data planes from a single control plane. This allows storage experts to reduce the number of compute resources required to process IO requests, reducing storage costs while increasing efficiency. \n \n\n**Containers on Micro-VM servers** \n\nMayastor volumes can be used to provide storage over vhost users, enabling storage teams to persist data for multi-tenant containers in virtual machines. Mayastor storage controllers can be deployed on lightweight virtual machines (MicroVMs) and function-based services to enable the efficient execution of IO tasks for serverless and container workloads. \n\n## Summary \n\nOpenEBS Mayastor is the first Container Attached Storage engine developed with the tremendous performance capabilities of the NVMe protocol itself in mind. Written in Rust and open source, Mayastor should be the preferred choice for workloads that need the ease of use and portability of OpenEBS plus the performance otherwise only delivered by the much harder to manage and to protect direct access to high performing disks and cloud volumes. \n \n\nTo know more on how Mayastor can help provision lightning fast storage solution for your dynamic workloads, drop us a message [here](https://openebs.io/community). Alternatively, you can also visit the official [documentation](https://mayastor.gitbook.io/introduction/quickstart/deploy-a-test-application) to learn more about Mayastor, its appropriate use-cases, and feature announcements. \n","notHasFeatureImage":false,"slug":"mayastor-1.0-has-arrived"},{"id":2,"title":"Install WordPress using OpenEBS Dynamic NFS Provisioner","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"02-11-2021","tags":["OpenEBS NFS","NFS Provisioner","RWX Volume Provisioning","Dynamic NFS Provisioner"],"excerpt":"Guide to deploy WordPress by using OpenEBS dynamic NFS Provisioner","content":"\n### Introduction\n In large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. Mayadata, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. This blog is mainly for users who require ReadWriteMany(RWX) for Kubernetes applications.\n\n## OpenEBS Dynamic NFS Provisioner\n OpenEBS Dynamic NFS Provisioner will enable you to dynamically provision ReadWriteMany(RWX/MultiNode ReaderWriter) volume on any type of Kubernetes Persistent Volumes which supports dynamic provisioning, RWX is achieved by launching NFS Server on the Persistent Volume. In simple words, dynamic-nfs-provisioner will expose ReadWriteOnce volume as ReadWriteMany volume.\n\n**Terminology:**\n\n ![NFS terminology](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-nfs-terminology.png)\n\n Few workloads in Kubernetes require ReadWriteMany type of volumes, for example, WordPress, Magento, etc. In this blog, I will walk through the steps to configure WordPress with a dynamic-nfs-provisioner.\n\n\n## Deploy WordPress by using dynamic NFS provisioner\n For deploying WordPress it is required to have dynamic NFS provisioner installed, So let’s first install dynamic NFS provisioner and then WordPress application.\n\n### Prerequisites for deploying dynamic NFS provisioner\n- Minimum Kuberentes Version 1.18\n- NFS Client must be installed on all nodes where the application pod schedules.\n For Ubuntu/Debian run below command to install nfs client:\n ```sh\n sudo apt install nfs-common\n ```\n- Storage provider(supports dynamic provisioning) to serve backend volume claim requests. In this blog, we are going to use OpenEBS localpv-hostpath as a backend storage provider.\n\n### Install NFS Provisioner\n NFS provisioner will dynamically provision NFS volumes, install NFS provisioner using below helm commands and it will deploy NFS provisioner with basic configurations, NFS provisioner also accepts [values.yaml](https://github.com/openebs/dynamic-nfs-provisioner/tree/develop/deploy/helm/charts#configuration)\n\n - To install NFS provisioner add repository and then install nfs-provisioner\n\n #### Add OpenEBS repo for installation\n - Use the below command to add helm repo for OpenEBS\n ```sh\n helm repo add openebs https://openebs.github.io/charts\n ```\n - Once repo has been added successfully, update helm repo using the following command:\n ```sh\n helm repo update\n ```\n\n #### Installing OpenEBS NFS Provisioner\n Once the helm repo addition is successful, run the `helm install` command as specified below, in this command, we are disabling the local-pv provisioner which doesn’t require for NFS Provisioner to work and configure __.nfsStorageClass.backendStorageClass__ corresponding to backend StorageProvider\n\n ```sh\n helm install openebs openebs/openebs -n openebs --create-namespace \\\n --set legacy.enabled=false \\\n --set ndm.enabled=false \\\n --set ndmOperator.enabled=false \\\n --set localProvisioner.enabled=false \\\n --set nfs-provisioner.enabled=true \\\n --set nfs-provisioner.nfsStorageClass.backendStorageClass=\n ```\n - After successful helm installation, you can view the openebs-nfs-provisioner and openebs-kernel-nfs StorageClass will get deployed.\n\n **Output**:\n\n ```sh\n [develop@develop ~]$ kubectl get po -n openebs\n NAME READY STATUS RESTARTS AGE\n openebs-nfs-provisioner-79b6ccd59-v8p6s 1/1 Running 0 4m12s\n\n [develop@develop~]$ kubectl get sc\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\n openebs-kernel-nfs openebs.io/nfsrwx Delete Immediate false 67s\n ```\n\n **Optional(installation of local-pv-provisioner)**\n\n The below command will install OpenEBS local-pv provisioner along with NFS provisioner, as stated above local-pv is used as backend StorageProvider in this blog\n\n ```sh\n helm install openebs openebs/openebs -n openebs --create-namespace \\\n --set legacy.enabled=false \\\n --set ndm.enabled=false \\\n --set ndmOperator.enabled=false \\\n --set localProvisioner.enabled=true \\\n --set localProvisioner.hostpathClass.name=openebs-hostpath \\\n --set localprovisioner.basePath=/var/openebs/hostpath \\\n --set nfs-provisioner.enabled=true \\\n --set nfs-provisioner.nfsStorageClass.backendStorageClass=openebs-hostpath\n ```\n\n **Note**: `localprovisioner.basePath` defines the custom hostpath directory to provide storage for backend PVCs.\n\n### Install WordPress\n\n [WordPress](https://en.wikipedia.org/wiki/WordPress) is one of the most famous blogging platforms and also it is bloggers first choice because of it's easy to handle nature. One can dynamically scale web servers of WordPress(content management systems [CMS]) by making use of Horizontal Pod Autoscaler feature of Kubernetes i.e increasing load will automatically scale the wordpress pods, users can install WordPress in multiple ways, but in this blog, we are going to install from its Helm repository by using [MariaDB](https://en.wikipedia.org/wiki/MariaDB) server to satisfy the requirements of WordPress server.\n\n #### Add helm repo for WordPress\n\n - Use the below command to add helm repo for WordPress\n ```sh\n helm repo add bitnami https://charts.bitnami.com/bitnami\n ```\n - Once repo has been added successfully, update helm repo using the following command\n ```sh\n helm repo update\n ```\n\n #### Installing WordPress\n Once the repo has been updated successfully, we can install WordPress referring to dynamic NFS provisioner storageclass. In this case storageclass will be openebs-kernel-nfs, if you have different storageclass then update the value of --set persistence.storageClass accordingly. WordPress also accepts various other parameters, for more information click [here](https://github.com/bitnami/charts/tree/master/bitnami/wordpress#parameters)\n\n ```sh\n helm install my-release -n wordpress --create-namespace \\\n --set wordpressUsername=admin \\\n --set wordpressPassword=password \\\n --set mariadb.auth.rootPassword=secretpassword \\\n --set mariadb.primary.persistence.enabled=true \\\n --set mariadb.primary.persistence.storageClass=openebs-hostpath \\\n --set mariadb.primary.persistence.accessModes={ReadWriteOnce} \\\n --set persistence.storageClass=openebs-kernel-nfs \\\n --set persistence.accessModes={ReadWriteMany} \\\n --set volumePermissions.enabled=true \\\n --set autoscaling.enabled=true \\\n --set autoscaling.minReplicas=2 \\\n --set autoscaling.maxReplicas=6 \\\n --set autoscaling.targetCPU=80 \\\n bitnami/wordpress\n ```\n\n Above helm installation should create following resources in the cluster:\n\n - Two WordPress application pod instances with RWX persistent volume.\n - One Mariadb application pod instance with RWO persistent volume.\n\n Results of helm wordpress installation:\n ```sh\n kubectl get po -n wordpress\n NAME READY STATUS RESTARTS AGE\n my-release-mariadb-0 1/1 Running 0 2m34s\n my-release-wordpress-85dc688bb9-24jx2 1/1 Running 0 2m34s\n my-release-wordpress-85dc688bb9-wkfpd 1/1 Running 1 2m18s\n ```\n\n Once the installation process is succeeded, follow the steps mentioned in output of _helm install_ command to access WordPress from your browser:\n\n ![WordPress login page](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-login-page.png)\n\n Now it's time to hack WordPress:\n\n ![WordPress landing page](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-landing-page.png)\n\n**Deployment View**:\n\n ![WordPress Deployment view](/images/blog/install-wordpress-using-dynamic-nfs-provisioner-deployment-view.jpg)\n\n\n **Background details on how NFS provisioner on provisioning RWX volume:**\n\n Once the PersistentVolumeClaim is created by referring to NFS provisioner storageclass, NFS provisioner will process the volume request and perform following actions:\n - Create Backend PVC and wait till the backend storage provider provides a volume.\n - Create NFS server deployment on backend PVC to expose RWO volumes as RWX volume via kernel NFS service.\n - Create NFS server kubernetes service which helps for other services to communicate with NFS server.\n - Create PersistentVolume(PV) for requested PersistentVolumeClaim.\n\n View of resources created after provisioning volume:\n\n ```sh\n kubectl get all -n openebs\n\n NAME READY STATUS RESTARTS AGE\n pod/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323-57645d68f8-hrvvm 1/1 Running 0 12m\n \n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n service/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 ClusterIP 10.100.164.113 2049/TCP,111/TCP 12m\n \n NAME READY UP-TO-DATE AVAILABLE AGE\n deployment.apps/nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 1/1 1 1 12m\n \n kubectl get pvc -n openebs\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n nfs-pvc-4ca805a6-6bd2-46cd-a845-d2aee2562323 Bound pvc-4bde2312-15a6-4b06-a309-069b47ce569c 10Gi RWO openebs-hostpath 14m\n ```\n\n### Benefits of using dynamic NFS Provisioner\n Dynamic NFS provisioner has various advantages, few of them are:\n - Provides configurable options to reduce recovery times(i.e maintaining HighAvailability of server)\n - **LeaseTime**: Lease time defines the renewal period(in seconds) for client connection.\n - **GraceTime**: Grace time defines the recovery period(in seconds) to reclaim locks.\n\n Recovery options can be configured via StorageClass annotations:\n ```yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-rwx\n annotations:\n openebs.io/cas-type: nfsrwx\n cas.openebs.io/config: |\n # LeaseTime defines the renewal period(in seconds) for client state\n - name: LeaseTime\n value: 30\n # GraceTime defines the recovery period(in seconds) to reclaim locks\n - name: GraceTime\n value: 30\n ...\n ...\n ...\n ```\n - Provides options to launch an NFS server on a specific set of nodes by using [node-affinity](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/tutorial/node-affinity.md#nfs-server-node-affinity) feature.\n - Supports [manual](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/tutorial/nfs-volume-resize.md) expansion of NFS(RWX) volume, for automation please track issue [#37](https://github.com/openebs/dynamic-nfs-provisioner/issues/37).\n - Provides an option to make NFS volume available only for [specific set of applications](https://github.com/openebs/dynamic-nfs-provisioner/blob/develop/docs/troubleshooting/non-root-application-accesing-nfs-volume.md#intro) which has matching permissions.\n","notHasFeatureImage":true,"slug":"install-wordpress-using-openebs-dynamic-nfs-provisioner"},{"id":3,"title":"OpenEBS 3.0 Release","author":"Kiran Mova","author_info":"Founder, Contributor and Maintainer of OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-09-2021","tags":["OpenEBS"],"excerpt":"Announcing OpenEBS 3.0 release - with ability to create 9 types of volumes. ","content":" \nWe released OpenEBS in early 2017 as one of the first solutions to adopt the Container Attached Storage pattern to deliver per workload storage in part by building upon Kubernetes. OpenEBS is now one of the most popular storage solutions for Kubernetes; our metrics indicate that usage of OpenEBS has increased significantly in the last three years, by approximately 7x, with the most popular workloads being those that are resilient including NoSQL DBs that often do not use shared storage. \n\nA lot of the growth in usage by OpenEBS is due to the overall growth in Kubernetes for stateful workloads. The [CNCF’s 2020 Survey](https://www.cncf.io/wp-content/uploads/2020/11/CNCF_Survey_Report_2020.pdf) highlighted 55% of surveyed organizations now use stateful applications in production. **The survey also highlighted MayaData / OpenEBS is in the top-5 list of popular storage solutions.**\n\nOver the years, OpenEBS has steadily evolved into a full stack storage software with swappable storage engines, related control plane components, and integrations into other cloud native projects. \n\nA quick summary:\n- **OpenEBS 1.0** was launched with the goal of simplifying the journey of users, who were onboarding Stateful workloads into Kubernetes. The initial release included: \n * A stable distributed block storage called “Jiva” based on the Longhorn engine. \n * An initial version of the CStor engine based on ZFS that was much more resilient to node failures and optimized for working with Hard Drives and RAID. \n * Also components like Node Disk Manager that helped to manage local storage on Kubernetes nodes and Dynamic LocalPV Provisioners.\n- **OpenEBS 2.0** focused on the stability of the data plane and building of an automated test framework, and the introduction of additional engines based on user feedback. This release included: \n * Making OpenEBS a CNCF project - which meant streamlining a lot of project governance, development processes and moving towards CNCF provided tools. \n * CStor data plane was declared stable, with support for incremental backup/restore support via Velero. \n * E2e tests for all of the OpenEBS Engines using the Litmus framework, which went on to become a project of its own.\n * ZFS LocalPV provisioners (CSI Driver) was introduced.\n * Initial release of Mayastor - which was the result of investigation into inherent performance limitations of earlier approaches including Jiva and CStor.\n- **OpenEBS 3.0** has been about enhancing the usability and stability aspects of all the previously introduced engines as well as support for new engines. The release is also about supporting the newer Kubernetes releases. This release includes:\n * CSI Drivers for CStor and Jiva and tools to migrate volumes to the newer CSI drivers. The legacy provisioners are deprecated with 3.0.0 and users are required to migrate to the corresponding CSI drivers asap. \n * Several enhancements to existing LocalPV flavors and introduction of new types of LocalPV. \n * A new control plane for Mayastor is being worked on that is better designed to handle scale and resiliency.\n * Also included are the initial version of a kubectl plugin for OpenEBS and a prometheus grafana mixin for managing the OpenEBS storage and volumes. \n\n## What’s new in OpenEBS 3.0?\n\nOpenEBS 3.0 (See [Release Notes](https://github.com/openebs/openebs/releases/tag/v3.0.0)) is a culmination of efforts geared towards laying the foundation for making it easier to onboard and accept community contributions, making each of the engine operators ready for future Kubernetes releases, making it easy to manage and troubleshoot various engines. This has been achieved via migration to latest Kubernetes constructs, ease of use improvements, bug fixes and most importantly refactoring the control plane and e2e test suites to independently enhance and release each of the engines. \n\nI highlight a few of these enhancements below. \n\nOpenEBS includes various storage engines developed for different workloads and use-cases. OpenEBS data engines can be classified into two categories - Local and Replicated.\n\n### LocalPV - New Features and Enhancements\n\nOpenEBS uses LocalPV provisioners to connect applications directly with storage from a single node. This storage object, known as LocalPV, is subject to the availability of the node on which it is mounted, making it a handy feature for fault-tolerant applications who prefer local storage over traditional shared storage. The OpenEBS LocalPV provisioner enables Kubernetes-based stateful applications to leverage several types of local storage features ranging from raw block devices to using capabilities of filesystems on top of those devices like LVM and ZFS. \n\nOpenEBS 3.0 includes the following enhancement to the LocalPV provisioner:\n* Support for 3 new types of LocalPVs namely LVM LocalPV, Rawfile LocalPV, Partition Local PV in addition to the previously supported Hostpath LocalPV, Device LocalPV and ZFS LocalPV. \n* [OpenEBS Hostpath LocalPV (stable)](https://github.com/openebs/dynamic-localpv-provisioner), the first and the most widely used LocalPV now supports enforcing XFS quota, ability to use a custom node label for node affinity (instead of the default `kubernetes.io/hostname`) \n* [OpenEBS ZFS LocalPV (stable)](https://github.com/openebs/zfs-localpv), used widely for production workloads that need direct and resilient storage has added new capabilities like:\n * Velero plugin to perform incremental backups that make use of the copy-on-write ZFS snapshots.\n * CSI Capacity based scheduling used with `waitForFirstConsumer` bound Persistent Volumes. \n * Improvements to inbuilt volume scheduler (used with `immediate` bound Persistent Volumes) that can now take into account the capacity and the count of volumes provisioned per node.\n* [OpenEBS LVM LocalPV (stable)](https://github.com/openebs/lvm-localpv), can be used to provision volume on top of LVM Volume Groups and supports the following features:\n * Thick (Default) or Thin Provisioned Volumes \n * CSI Capacity based scheduling used with `waitForFirstConsumer` bound Persistent Volumes. \n * Snapshot that translates into LVM Snapshots \n * Ability to set QoS on the containers using LVM Volumes.\n * Also supports other CSI capabilities like volume expansion, raw or filesystem mode, metrics. \n* [OpenEBS Rawfile LocalPV (beta)](https://github.com/openebs/rawfile-localpv), is a preferred choice for creating local volumes using a sparse file within a sub-directory that supports capacity enforcement, filesystem or block volumes.\n* [OpenEBS Device LocalPV (beta)](https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md#provisioning-localpv-device-persistent-volume), is a preferred choice for running workloads that have typically worked well with consuming the full disks in block mode. This provisioner uses NDM to select the block device. \n* [OpenEBS Partition LocalPV (an alpha engine)](https://github.com/openebs/device-localpv), is under active development and is being deployed in select users for creating volumes by dynamically partitioning a disk with the requested capacity from the PVC. \n\n### ReplicatedPV - New Features and Enhancements\n\nOpenEBS uses ReplicatedPV provisioners to connect applications to volumes - whose data is synchronously replicated to multiple storage nodes. This storage object, known as ReplicatedPV, is highly available and can be mounted from multiple nodes in the clusters. OpenEBS supports three types of ReplicatedPVs Jiva (based on Longhorn and iSCSI), CStor (based on ZFS and iSCSI) and Mayastor (based on SPDK and NVMe). \n\nSome enhancements to replicated storage engines in OpenEBS 3.0 include:\n* [OpenEBS Jiva (stable)](https://github.com/openebs/jiva-operator), has added support for a CSI Driver and Jiva operator that include features like:\n * Enhanced management of the replicas \n * Ability to auto-remount the volumes marked as read-only due to iSCSI time to read-write. \n * Faster detection of the node failure and helping Kubernetes to move the application out of the failed node to a new node. \n * 3.0 also deprecates the older Jiva volume provisioners - that was based on the kubernetes external storage provisioner. There will be no more features added to the older provisioners and users are requested to migrate their Volumes to CSI Drivers as soon as possible. \n* [OpenEBS CStor (stable)](https://github.com/openebs/cstor-operators), has added support for a CSI Driver and also improved customer resources and operators for managing the lifecycle of CStor Pools. This 3.0 version of the CStor includes:\n * The improved schema allows users to declaratively run operations like replacing the disks in mirrored CStor pools, add new disks, scale-up replicas, or move the CStor Pools to a new node. The new custom resource for configuring CStor is called CStorPoolCluster (CSPC) compared to older StoragePoolCluster(SPC). \n * Ability to auto-remount the volumes marked as read-only due to iSCSI time to read-write. \n * Faster detection of the node failure and helping Kubernetes to move the application out of the failed node to a new node. \n * 3.0 also deprecates the older CStor volume provisioners and pool operators based on SPC - that was based on the kubernetes external storage provisioner. There will be no more features added to the older provisioners and users are requested to migrate their Pools to CSPC and Volumes to CSI Drivers as soon as possible.\n* [OpenEBS Mayastor (beta)](https://github.com/openebs/mayastor), is under active development and currently is building a new and enhanced control plane to manage the mayastor pools and volumes. In the current release, the changes to the Mayastor include:\n * Support for deprecating the MOAC based control plane in favor of the new control plane. \n * Enhanced control plane to handle node failure scenarios and move the volumes to new nodes. \n * Stabilizing the Mayastor data engine for durability and performance. \n * Enhanced E2e tests.\n\n### Other Notable Features and Enhancements\n\nBeyond the improvements to the data engines and their corresponding control plane, there are several new enhancements that will help with ease of use of OpenEBS engines:\n* Several fixes and enhancements to the Node Disk Manager like automatically adding a reservation tag to devices, detecting filesystem changes and updating the block device CR (without the need for a reboot), metrics exporter and an API service that can be extended in the future to implement storage pooling or cleanup hooks. \n* [Dynamic NFS Provisioner](https://github.com/openebs/dynamic-nfs-provisioner) that allows users to launch a new NFS server on any RWO volume (called backend volume) and expose an RWX volume that saves the data to the backend volume. \n* Kubernetes Operator for automatically upgrading Jiva and CStor volumes that are driven by a Kubernetes Job \n* Kubernetes Operator for automatically migrating CStor Pools and Volumes from older pool schema and legacy (external storage based) provisioners to the new Pool Schema and CSI volumes respectively. \n* [OpenEBS CLI (a kubectl plugin)](https://github.com/openebs/openebsctl) for easily checking the status of the block devices, pools (storage) and volumes (PVs). \n* [OpenEBS Dashboard](https://github.com/openebs/monitoring) (a prometheus and grafana mixin) that can be installed via jsonnet or helm chart with a set of default Grafana Dashboards and AlertManager rules for OpenEBS storage engines. \n* Enhanced OpenEBS helm chart that can easily enable or disable a data engine of choice. The 3.0 helm chart stops installing the legacy CStor and Jiva provisioners. If you would like to continue to use them, you have to set the flag “legacy.enabled=true”. \n* OpenEBS helm chart includes sample kyverno policies that can be used as an option for PodSecurityPolicies(PSP) replacement. \n* OpenEBS images are delivered as multi-arch images with support for AMD64 and ARM64 and hosted on DockerHub, Quay and GHCR. \n* Support for installation in air gapped environments. \n* Enhanced Documentation and Troubleshooting guides for each of the engines located in the respective engine repositories. \n* A new and improved design for the [OpenEBS website](https://openebs.io/). \n\n### Deprecated Features\n\nAs announced in [June earlier this year](https://github.com/openebs/openebs/releases/tag/v2.10.0), the non-csi provisioners for CStor and Jiva are deprecated with 3.0. You can still continue to use them till Dec 2021. The older provisioners are released with the v2.12.2 version at the moment and only patch releases (to fix severe security vulnerabilities) will be supported going forward. \n\n## What is Next after OpenEBS 3.0? \n\nWe originally developed OpenEBS that focused on solving the core issue of Kubernetes storage - that is the lack of support to manage stateful applications, especially modern resilient workloads that do not use shared storage and prefer the per workload approach of Container Attached Storage. That remains our focus today - and as mentioned innovation seems to be accelerating as the community of users and contributors grows at an accelerated rate. \nI am especially thankful to the OpenEBS users and organizations that have taken the time to fill out [Adopters.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md) to share how they are using OpenEBS, and how it is solving their common problems while leveraging Kubernetes for data. \n\nI will also be talking about the new features, upgrades & bug fixes and giving glimpses into what is coming in OpenEBS 3.1 and 4.0 at the CNCF On-Demand Webinar: OpenEBS 3.0 : What’s in it on September 30th. Feel free to register & attend: [https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-openebs-30-whats-in-it/](https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cncf-on-demand-webinar-openebs-30-whats-in-it/)\n\nTo learn more about recent updates, developer documentation, and scheduled releases, please feel free to use any of the following resources.\n- [https://github.com/openebs/openebs/releases](https://github.com/openebs/openebs/releases)\n- [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md)\n- [https://openebs.io/docs](https://openebs.io/docs)\n- [https://openebs.io/community](https://openebs.io/community)\n\n\n\n\n\n\n\n\n\n\n\n\n\n","notHasFeatureImage":true,"slug":"openebs-3.0-release"},{"id":4,"title":"A hands on demo of Volume Populator using OpenEBS LVM CSI driver","author":"Shovan Maity","author_info":"Shovan works as a Software Engineer at MayaData, who's experienced in Load Balancer, gRPC, WebSocket, REST APIs, and has good hands-on experience on Kubernetes. In his free time, Shovan likes to read blogs on distributed systems. He also likes Travelling and Photography.","date":"05-07-2021","tags":["OpenEBS","LVM CSI driver"],"excerpt":"In this blog we will write a volume populator and create a volume with that populator. This is written using lib-volume-populator. This is a control loop on PVC that maintains the lifecycle of the PV and PVC for volume populator.","content":"\nIn this blog we will write a volume populator and create a volume with that populator. Source code of this volume populator is available [here](https://github.com/shovanmaity/s3-populator). This is written using [lib-volume-populator](https://github.com/kubernetes-csi/lib-volume-populator). This is a control loop on PVC that maintains the lifecycle of the PV and PVC for volume populator. If we want to write a volume populator we need to write a CRD spec like [this](https://github.com/shovanmaity/s3-populator/blob/main/types.go) and we need to code on how to write the data on the volume. We can build or reuse any plugin that can write data from source to the volume. Here we will use [s3 sync](https://github.com/shovanmaity/s3-sync) image to write data from s3 to local volume. This can copy data from a bucket or a particular directory in a bucket. This may have some other use cases. In this example we will create, build a react app, push it to MinIo and then we will use that s3 bucket to create volumes for nginx instances.\n\nIn this exercise we will use the OpenEBS LVM CSI driver though it will work with any CSI driver and we will perform this demo in a local minikube cluster. The steps are given below. \n\n* Create a minikube cluster with AnyVolumeDataSource alpha feature enabled mode.\n \n ```\n minikube start --feature-gates=AnyVolumeDataSource=true\n ```\n\n* Install OpenEBS LVM CSI driver.\n \n ```\n kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/lvm-operator.yaml\n ```\n\n* Create PV and VG in the host machine.\n \n ```\n truncate -s 10G /tmp/disk.img\n sudo losetup -f /tmp/disk.img --show\n #get loop device id and replace X with it\n sudo pvcreate /dev/loopX\n sudo vgcreate lvmvg /dev/loopX\n ```\n\n* Create a storage class to create lvm volumes using this volume group.\n\n ``` \n kubectl create -f - << EOF\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-lvmpv\n parameters:\n storage: lvm\n volgroup: lvmvg\n provisioner: local.csi.openebs.io\n EOF\n ```\n\n* Install volume populator controller.\n\n ``` \n kubectl apply -f https://raw.githubusercontent.com/shovanmaity/s3-populator/main/crd.yaml\n kubectl apply -f https://raw.githubusercontent.com/shovanmaity/s3-populator/main/deploy.yaml\n ```\n\n* Run MinIO on the local machine as a S3 source.\n\n ```\n docker run -p 9000:9000 \\\n -e \"MINIO_ROOT_USER=minioadmin\" \\\n -e \"MINIO_ROOT_PASSWORD=minioadmin\" \\\n minio/minio server /data\n ```\n\n* Create and build a react app in local machine.\n \n ```\n npx create-react-app my-app\n cd my-app\n npm build\n ```\n\n* Create a S3 bucket to push the static files.\n \n ```\n AWS_ACCESS_KEY_ID=minioadmin \\\n AWS_SECRET_ACCESS_KEY=minioadmin \\\n AWS_REGION=us-east-1 \\\n aws --endpoint-url http://192.168.0.190:9000 s3api create-bucket --bucket my-bucket\n ```\n\n* Make sure we are inside the build folder of the react app and then push the static files to s3.\n \n ```\n AWS_ACCESS_KEY_ID=minioadmin \\\n AWS_SECRET_ACCESS_KEY=minioadmin \\\n AWS_REGION=us-east-1 \\\n aws --endpoint-url http://192.168.0.190:9000 s3 cp . s3://my-bucket --recursive\n ```\n\n* Create a S3 populator cr with S3 and bucket related information.\n\n ```\n kubectl create -f - << EOF\n apiVersion: example.io/v1\n kind: S3Populator\n metadata:\n name: s3-populator-1\n namespace: default\n spec:\n url: http://192.168.0.190:9000\n id: minioadmin\n secret: minioadmin\n region: us-east-1\n bucket: my-bucket\n key: /\n EOF\n ```\n\n* Create a nginx service to access it.\n \n ```\n kubectl create -f - << EOF\n apiVersion: v1\n kind: Service\n metadata:\n name: nginx\n namespace: default\n labels:\n app: nginx\n spec:\n ports:\n - port: 80\n name: web\n type: NodePort\n selector:\n app: nginx\n EOF\n ```\n\n* Create a nginx sts.\n \n ```\n kubectl create -f - << EOF\n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: web\n namespace: default\n spec:\n serviceName: nginx\n replicas: 1\n selector:\n matchLabels:\n app: nginx\n template:\n metadata:\n labels:\n app: nginx\n spec:\n containers:\n - name: nginx\n image: registry.k8s.io/nginx-slim:0.8\n ports:\n - containerPort: 80\n name: web\n volumeMounts:\n - name: www\n mountPath: /usr/share/nginx/html\n volumeClaimTemplates:\n - metadata:\n name: www\n spec:\n dataSource:\n apiGroup: example.io\n kind: S3Populator\n name: s3-populator-1\n storageClassName: \"openebs-lvmpv\"\n accessModes: [ \"ReadWriteOnce\" ]\n resources:\n requests:\n storage: 500Mi\n EOF\n ```\n\n* Get the service IP and port and open it in any browser. We will be able to see the same content(react app) in our nginx app.\n\nPlease join our community if you have any feedback or queries on the above demo and OpenEBS.","notHasFeatureImage":false,"slug":"a-hands-on-demo-of-volume-populator-using-openebs-lvm-csi-driver"},{"id":5,"title":"Life cycle of PVC with Volume Populator","author":"Shovan Maity","author_info":"Shovan works as a Software Engineer at MayaData, who's experienced in Load Balancer, gRPC, WebSocket, REST APIs, and has good hands-on experience on Kubernetes. In his free time, Shovan likes to read blogs on distributed systems. He also likes Travelling and Photography.","date":"05-07-2021","tags":["PVC","Volume Populator"],"excerpt":"This blog is on the life cycle of pvc with a volume populator. Volume populator feature is in alpha in Kubernetes (v1.21 release). Creating a volume with base or seed data is one of the requirements in storage.","content":"\nThis blog is on the life cycle of pvc with a [volume populator](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators). Volume populator feature is in alpha in Kubernetes (v1.21 release). Creating a volume with base or seed data is one of the requirements in storage. In Kubernetes 1.12, the _DataSource_ field is added to the PVC spec. The field was implemented as a _TypedLocalObjectReference_ to give flexibility in the future about what objects could be data sources for new volumes. Since then, it supports only two things to be the source of a new volume - existing PVC(clone volume) and snapshots(restore a snapshot). Implementation of these two data sources relies on the CSI plugin to do the actual work. As it can point to any CR in Kubernetes validation on the DataSource field of PVC is relaxed and relies on a new async controller to perform the validation on those data sources and provide feedback to users. A new CRD is introduced to register valid dataSource, and we need to create a resource for individual volume populators to inform that the volume populator is a valid dataSource for a PVC.\n\nIn the first part we will see the volume lifecycle without a volume populator. In this process involved components are _**API server**, **etcd**, **CSI external provisioner**_ and _**CSI controller plugin**._ We will not go into the CSI node plugin, we will stop as soon as PVC is in bound state. Let's understand the responsibility of some modules present here.\n\n![volume lifecycle without a volume populator](https://lh6.googleusercontent.com/rX9n7NSjDPiwL0xf1YR0hfOF0pf5TY7qgzMrKhhQyARxREFOUakXS0aAWFZ4Y1MKE7b6tUZ33iYVidZdPyOXPhgW0G60lmIGZOz5KadBJaQirYK6pjTWUix2Hvk4laXXzFPfg803)\n\n**CSI external provisioner**\n\nThis is a CSI driver sidecar component. It is a control loop on PVC. As clone and restore volumes come under CSI workflow if _DataSource_ is present and it is PVC or snapshot then only it takes some actions for that PVC else it will ignore that PVC.\n\n**CSI controller plugin**\n\nThis is a CSI driver sidecar component provided by the storage provider(SP). It is a gRPC server which implements CSI controller services. CSI external provisioner makes a CreateVolume gRPC call to it once new PVC is created.\n\n* User creates a pvc and it is written in etcd with the help of API server and the pvc goes to the pending state.\n* CSI external provisioner observes that pvc and makes _CreateVolume_ call to the CSI controller plugin sidecar. On successful response it creates a PV for that PVC and applies bi-directional reference between them.\n* Once bi-directional reference is done, pvc goes to bound state.\n\nNow we will see the volume lifecycle with the data populator. Involved components are _**API server**, **etcd**, **CSI external provisioner**, **CSI controller plugin** and **Volume Populator controller**._ We will not go into the CSI node plugin, we will stop as soon as PVC is in bound state. Let's understand the responsibility of some modules present here. \n\n![volume lifecycle with the data populator](https://lh4.googleusercontent.com/-uuZ8z1w6y-jGSOomfn_VuIQt_6nOtOUv11fS1Qac0p7FhneZMM617gfjf49ek-YKiKVHpEYqbc2787UhIvxq7lTeVm4OKCsMFwdy_iZ-fNpzrgqaU4RP2864LNlcTC0JWKCeXoF)\n\n**Volume Populator Controller**\n\nIt is a control loop on PVC, responsible to manage PVC pointing to a particular dataSource. Namespace in which this controller is deployed is reserved for creating some intermediate PVC and pod to populate the data in the volume. If we create a PVC in that namespace for our application with a valid dataSource it will not work, this control loop will skip that PVC. This controller adds a finalizer to the initial PVC. It is written using [lib volume populator](https://github.com/kubernetes-csi/lib-volume-populator). [This](https://github.com/kubernetes-csi/lib-volume-populator/tree/master/example/hello-populator) is an example of a data populator written using lib volume populator. One new CRD _Volume Populator_ is introduced_._ Like CSI drivers we have to register the volume populator.\n\n* User creates a PVC with dataSource pointing to a volume populator and it is written in etcd with the help of API server and the PVC goes to the pending state.\n* CSI external provisioner observes that PVC and takes no action as the dataSource is not pointing to a PVC or snapshot.\n* Volume populator controller is another watcher on pvc. It creates a PVC’ which is the same as the PVC created by the user without the dataSource field.\n* As PVC’ does not have a dataSource CSI external provisioner makes a CreateVolume gRPC call to the CSI controller plugin. Once the gRPC call is successful PV’ is created and bi-directional reference applied on PVC’ and PV’. Then PVC’ goes to the bound state.\n* Once PVC’ is in bound state, the volume populator controller creates a pod to populate the data for that PVC’.\n* Once data is populated it patches PVC and PV’ and applies a bi-directional reference between  them. After this PVC will be in bound state and PVC’ will be in lost state. Then the populator pod and PVC’ are deleted by this controller.\n\nAs mentioned above in core Kubernetes validation on the DataSource field is relaxed now we can create PVC pointing to any data source. To add validation around it one new asynchronous PVC [controller](https://github.com/kubernetes-csi/volume-data-source-validator) is added. This controller is responsible for posting warning events if PVC is pointing to invalid DataSource. One new CRD of kind VolumePopulator and API version populator.storage.k8s.io/v1alpha1 is introduced to specify a valid volume populator.\n\nHere is one valid data source -\n\n```\n kind: VolumePopulator\n apiVersion: populator.storage.k8s.io/v1alpha1\n metadata:\n name: valid-populator\n sourceKind:\n group: valid.storage.k8s.io\n kind: Valid\n```\n\nHere is an example PVC with valid data source -\n\n```\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: valid-pvc-datasource\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 1Gi\n dataSource:\n apiGroup: valid.storage.k8s.io\n kind: Valid\n name: valid\n```\n\nHere is an example of PVC with invalid data source -\n\n```\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: invalid-pvc-datasource\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 1Gi\n dataSource:\n apiGroup: invalid.storage.k8s.io\n kind: Invalid\n name: invalid\n```\n\nEvent in invalid data source PVC-\n\n```\n shovan@probot:~$ kubectl get event | grep Warning\n 58s Warning UnrecognizedDataSourceKind persistentvolumeclaim/invalid-pvc-datasource The data source for this PVC does not match any registered VolumePopulator\n\n shovan@probot:~$ kubectl describe pvc invalid-pvc-datasource\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal ExternalProvisioning 115s (x2 over 115s) persistentvolume-controller waiting for a volume to be created, either by external provisioner \"k8s.io/minikube-hostpath\" or manually created by system administrator\n Normal Provisioning 114s k8s.io/minikube-hostpath_minikube_e7c97d0c-eb52-42bc-af55-d8a167ab3987 External provisioner is provisioning volume for claim \"default/invalid-pvc-datasource\"\n Normal ProvisioningSucceeded 113s k8s.io/minikube-hostpath_minikube_e7c97d0c-eb52-42bc-af55-d8a167ab3987 Successfully provisioned volume pvc-7df410d5-1efe-4d69-b481-57cbde9f10cd\n Warning UnrecognizedDataSourceKind 33s (x6 over 114s) data-source-validator The data source for this PVC does not match any registered VolumePopulator\n```\n\nPlease join our [community](https://openebs.io/community) if you have any feedback or queries on the article or anything related to [OpenEBS](https://openebs.io/)\n","notHasFeatureImage":false,"slug":"life-cycle-of-pvc-with-volume-populator"},{"id":6,"title":"Logical Volume Management (LVM) on Kubernetes with OpenEBS LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"17-06-2021","tags":["Kubernetes","OpenEBS","LocalPV","LVM"],"excerpt":"In large production environments, various physical disks are typically pooled to handle user and application storage needs. Logical Volume Management (LVM) is a storage virtualization technology that is a part of Linux which is widely used as a way to help administrators and developers abstract the physical configuration of storage devices for flexible and simpler provisioning. ","content":"\n## Introduction to Logical Volume Management\n\nIn large production environments, various physical disks are typically pooled to handle user and application storage needs. Logical Volume Management (LVM) is a **storage virtualization** technology that is a part of Linux which is widely used as a way to help administrators and developers **abstract the physical configuration** of storage devices for flexible and simpler provisioning. The LVM tool gathers all storage devices into volume groups, creates logical units depending on application needs, and then allocates them to particular groups. By doing so, LVM gives greater control, flexibility and easier abstraction of physical storage enabling use cases such as the replacement of local devices, local RAID and the ability to manage the sharing of JBODs across multiple hosts.\n\nOpenEBS offers a **LVM option for LocalPV** that allows Kubernetes administrators to dynamically provision Persistent Volumes. This article delves into the architecture of Logical Volume Management and how to set up and use the **OpenEBS LVM LocalPV** to provision and manage storage for a Kubernetes cluster.\n\n## LVM Architecture & Concepts\n\nAt its foundation, LVM combines several individual physical disks into one Volume Group that can be subdivided into multiple Logical Volumes. This is achieved by creating several layered abstractions between the physical drives and the end-user’s storage partitions.\n\n![A typical PVC user request flow](/images/blog/a-typical-pvc-user-request-flow.png) \n(***A typical PVC user request flow***)\n\nThe various storage management structure that LVM uses to simplify the management of conflicting end-user storage needs, in increasing order of virtualization are: \n\n## Physical Volumes\n\nThese are regular storage devices that include physical disks such as HDDs, SSDs and RAIDs. Such units are essentially used as basic raw devices for achieving a level of abstraction. To manage a physical disk, the LVM assigns a header to it which it uses while allocating volumes.\n\n### Volume Groups (VG)\n\nVolume Groups are LVM’s higher level of abstraction and consist of one or more physical volumes. Once a physical storage device is added into a VG, the files stored inside it are partitioned into multiple smaller logical **extents** as specified by administrators when creating the Volume Group. Besides partition size, administrators can also set a VG’s security permissions and other specifications.\n\n### Logical Volumes\n\nA Volume Group (VG) is partitioned into several Logical Volumes (LVs) that can be perceived as flexible partitions of volume. LVs are the primary abstract interface through which end-users and applications interact with storage. \n\nVolumes in a Volume Group are partitioned into smaller segments known as **Extents**. While Physical volumes are divided into **Physical Extents**, Logical Volumes are divided into **Logical Extents**. An extent size is the smallest amount of space that the LVM can allocate since the Logical Volumes are just abstract mapping between physical and logical extents.\n\nExtents give LVM flexibility since logical extents do not need to be directly and continuously mapped to physical extents. An LVM can duplicate and rearrange the physical extents in a physical volume without changing logical extents, eliminating the need to migrate users. This gives an efficient way of resizing logical extents by adding or removing physical extents from the volume.\n\n### OpenEBS LVM LocalPV\n\nThe OpenEBS Container Storage Interface (CSI) driver allows Kubernetes administrators to dynamically resize Local Persistent Volumes using Logical Volume Management. There are several capabilities of OpenEBS LocalPV LVM that readers are encouraged to learn. These are available on both OpenEBS docs or via the published [OpenSource roadmap](https://github.com/orgs/openebs/projects/30). OpenEBS LocalPV LVM is a widely deployed project, which is under active development with the assistance of the broader OpenEBS community, including the hyper-scale eCommerce firm Flipkart. In the section below, we cover the steps to install a OpenEBS LVM driver in a Kubernetes cluster, along with highlighting its possible use-cases.\n\n## Installing OpenEBS LVM into a Kubernetes Cluster\n\n### Setup\n\nFor the CSI LVM driver to work on a cluster, it must meet the following prerequisites:\n\n1. All nodes should have LVM2 utilities installed on their host systems. Additional details on LVM2 utilities can be found here: \n\n[Ubuntu - http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html](http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html)\n\n[Red Hat - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_logical_volumes/index](http://manpages.ubuntu.com/manpages/xenial/man8/lvm.8.html).\n\n2. A Volume Group of block storage devices has been set up to provision volume for the cluster. \n\n3. The user has administrative rights so they can install RBAC components into the default namespace for objects created by the Kubernetes System: **kube-system**.\n\nThe OpenEBS CSI LVM driver supports Kubernetes 1.18 and newer versions, the Ubuntu OS and runs on LVM2 utilities. \n\nIf you have a real physical device, you can use the command accordingly or can use a loopback device for testing. The following commands are used to locate the disk to be managed by LVM.\n\n $ truncate -s 1024G /tmp/disk.img //marks the extents\n $ sudo losetup -f /tmp/disk.img --show //finds unused device\n\nThe Volume Group is created on all nodes using commands similar to the following:\n\n $ sudo pvcreate /dev/sdb\n $ sudo vgcreate lvmvg /dev/sdb\n\nThis creates the Volume Group `lvmvg`. The same process must be followed to create Volume Groups on all nodes. \n\n### Installation\n\nThe OpenEBS CSI LVM driver can then be installed from the official repository by running the command:\n\n $ kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/lvm-operator.yaml\n\nThe following command can be used to check whether the driver and its components have been installed: \n\n $ kubectl get pods -n kube-system -l role=openebs-lvm\n\nOnce the driver has been successfully installed, administrators can now use it to provision volumes through the steps to follow below.\n\n### Provisioning Volumes with the LVM Driver\n\nProvisioning Volumes with the OpenEBS LVM driver consists of three steps:\n\n1. Creating a storage class\n2. Creating a Persistent Volume Claim (PVC)\n3. Deploying the Application\n\nThe manifest file for a storage class that allows the LVM driver to create volumes on specific nodes is as shown:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: darwin-lvmpv\n allowVolumeExpansion: true\n parameters:\n volgroup: \"lvmvg\"\n provisioner: local.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: kubernetes.io/hostname\n values:\n - lvmpv-node1\n - lvmpv-node2\n\nThe YAML file above instructs the LVM to deploy volumes from the Volume Group `lvmvg` to nodes `lvmpv-node1` and `lvmpv-node2`.\n\nThe Persistent Volume Claim for this storage class will look similar to:\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: pvc-lvmpv\n spec:\n storageClassName: darwin-lvmpv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nThe application can now be allocated storage resources from the `darwin-vg` Volume Group.\n\n## OpenEBS LocalPV LVM Use-Cases\n\nOnce the LVM driver has been installed in a cluster, it allows dynamic provisioning of LocalPV Volumes to Kubernetes applications. This is useful for applications such as:\n\n* Dynamic resizing of assigned physical and logical volumes\n* Creating smaller storage environments for tests and upgrades\n* Tenant Isolation\n* Backup and Recovery using redundant partitions \n\nBy simplifying the management of users’ conflicting storage needs by abstracting logical and physical volumes, Logical Volume Management (LVM) is critical for managing production workloads efficiently. To do so, OpenEBS through a Container Storage Interface (CSI) Driver allows Kubernetes administrators to dynamically resize Local Persistent Volumes using Logical Volume Management. This enables an application-level **storage virtualization layer** for **greater control**, **flexibility** and **easier abstraction** of physical storage.","notHasFeatureImage":true,"slug":"logical-volume-management-(lvm)-on-kubernetes-with-openebs-localpv"},{"id":7,"title":"Provisioning OpenEBS Jiva volumes via CSI","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"09-06-2021","tags":["CSI","OpenEBS","Jiva"],"excerpt":"Container Storage Interface (CSI), is the new model for integrating storage system drivers with container orchestration systems like Kubernetes. This new interface is a major benefit to the container ecosystem as it standardizes the model of integrating storage systems with container orchestration systems.","content":"\n[Container Storage Interface (CSI)](https://github.com/container-storage-interface/spec), is the new model for integrating storage system drivers with container orchestration systems like [Kubernetes](https://kubernetes.io/). This new interface is a major benefit to the container ecosystem as it standardizes the model of integrating storage systems with container orchestration systems. Specifically for Kubernetes, it frees the storage system driver from being tied to the Kubernetes release schedule due to it being incorporated in the same code base. With CSI, storage system drivers can now be installed asynchronously to container orchestration releases, providing faster bug fixes and features.\n\nThe Openebs Jiva CSI driver provides a CSI interface that allows it to manage the lifecycle of OpenEBS Jiva volumes for persistent volumes. OpenEBS has recently released a beta version of the [Jiva CSI driver](https://github.com/openebs/jiva-operator) for one of its Container Attached Storage engines - [Jiva](https://github.com/openebs/jiva).\n\nAs of 2.7.0, the supported features are volume provisioning, de-provisioning, volume resize, raw block volume, and volume usage metrics. There are other key differences between the external-provisioner and CSI implementations, are benefits to users, like\n\n- Auto remount\n- Protecting against nodes mounting an empty directory accidentally and re-writing. \n- Fix for multi-attach errors on onprem k8s clusters\n- Standardised with CSI and further enhancements coming in K8s\n\nThis blog will demonstrate how to configure and consume Jiva volumes using the CSI Driver. \n\n## Prerequisites\n\n1. Kubernetes version 1.18 or higher\n2. iSCSI initiator utils installed on all the worker nodes\n3. You have access to install RBAC components into the openebs namespace.\n4. For resize feature, verify ExpandCSIVolumes and ExpandInUsePersistentVolumes feature gates are enabled on the kubelet of all worker nodes and kube-apiserver.\n\n## Installation\n\n### Using Jiva operator: \n\n1. Install openebs local PV provisioner to create hostpath based volumes.\n\n```\nkubectl apply -f https://openebs.github.io/charts/openebs-operator-lite.yaml\n```\n\n2. Install the latest Jiva-operator release.\n\n```\n$ kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml\n```\n\n3. Using Jiva Helm chart:\n\n```\n$ helm repo add openebs-jiva https://openebs.github.io/jiva-operator\n$ helm repo update \n$ helm install openebs-jiva/jiva\n```\n\nMore info related to Jiva charts, its configurable parameters, and their default values can be found [here](https://github.com/openebs/jiva-operator/tree/master/deploy/helm/charts).\n\n## Configure Jiva CSI Driver\n\n### OpenEBS Jiva CSI driver comprises of 2 components:\n\n1. A controller component launched as a StatefulSet, implementing the CSI controller services. The Control Plane services are responsible for creating/deleting the required OpenEBS Volume.\n2. A node component that runs as a DaemonSet, implementing the CSI node services. The node component is responsible for performing the iSCSI connection management and connecting to the OpenEBS Volume.\n\nThe node components make use of the host iSCSI binaries for iSCSI connection management. Depending on the OS, the spec will have to be modified to load the required iSCSI files into the node pods.\n\nVerify that the OpenEBS CSI Components are installed, one node service pod comes up for each node:\n\n $ kubectl get pods -n openebs -l role=openebs-jiva-csi\n\n NAME \t READY STATUS\tRESTARTS AGE\n openebs-csi-controller-0 6/6 \t Running 0 \t 6m14s\n openebs-csi-node-56t5g 2/2 \t Running 0 \t 6m13s\n\n## Before you provision a volume:\n\n**1. (Optional) Create JivaVolumePolicy (jvp)**\n\nCreate Jiva volume policy to configure various tunables required in case of day 2 operations, for example, replicationFactor (defaults to 3), resource limits, tolerations, node-selector, etc.\n\nAlthough It's not mandatory to create JivaVolumePolicy until the user wants to tune some specific default volume policies. If not provided, jiva operator uses the default policies.\n\n apiVersion: openebs.io/v1alpha1\n kind: JivaVolumePolicy\n metadata:\n name: example-volume-policy\n namespace: openebs\n spec:\n replicaSC: openebs-hostpath\n enableBufio: false\n autoScaling: false\n target:\n replicationFactor: 1\n\n $ kubectl get jvp -n openebs\n NAME AGE\n example-volume-policy 1m\n\n**2. Create Storage Class**\n\nCreate Storage to dynamically provision volumes using Jiva CSI driver with volume policy created above.\n\nRead more about various [jiva volume policies](https://github.com/openebs/jiva-operator/blob/master/docs/tutorials/policies.md).\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-jiva-csi-sc\n provisioner: jiva.csi.openebs.io\n allowVolumeExpansion: true\n parameters:\n cas-type: “jiva” \n policy: \"jiva-policy\" // optional\n\n## Volume Provisioning with FIO App Deployment\n\nRun your application by specifying the above Storage Class for the PVCs. For example, we will use the FIO application for demonstration purpose:\n\n**1. Create PVC:**\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: jiva-csi-demo\n spec:\n storageClassName: openebs-jiva-csi-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n\n**2. Deploy FIO Application**\n\nWhen using Pods with PersistentVolumes, we recommend that you use a workload controller (such as a Deployment or StatefulSet). While you would not typically use a standalone Pod, the following example uses one example deployment for simplicity.\n\nThe following example consumes the volume that you created in the previous section:\n\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: fio\n spec:\n selector:\n matchLabels:\n name: fio\n replicas: 1\n strategy:\n type: Recreate\n rollingUpdate: null\n template:\n metadata:\n labels:\n name: fio\n spec:\n containers:\n - name: perfrunner\n image: openebs/tests-fio\n command: [\"/bin/bash\"]\n args: [\"-c\", \"while true ;do sleep 50; done\"]\n volumeMounts:\n - mountPath: /datadir\n name: fio-vol\n volumes:\n - name: fio-vol\n persistentVolumeClaim:\n claimName: jiva-csi-demo\n\nWhile the asynchronous handling of the Volume provisioning is in progress, the application pod may throw some errors propagated from CAS pods to the application pods:\n\n* Waiting for JivaVolume to be Ready: Implies volume components are still being created\n\n* Volume is not ready: Replicas yet to connect to controller: Implies volume components are already created but yet to interact with each other.\n\nVerify that the FIO application pod is in running state:\n\n $ kubectl get pods\n\n NAME READY STATUS RESTARTS AGE\n Fio 1/1 Running 0 97s\n\n## Volume Expansion\n\n**Notes:**\n\n* Only dynamically provisioned volumes can be resized.\n* Volumes containing a filesystem of `XFS`, `Ext3`, `Ext4` type, or even raw block volume can be resized.\n\n**Steps:**\n\n1. Update the increased PVC size in the pvc spec section\n\n kind: PersistentVolumeClaim\n metadata:\n name: jiva-csi-demo\n spec:\n resources:\n requests:\n storage: 10Gi\n volumeName: pvc-136b015f-dac3-11e9-8980-42010a80006c\n status:\n accessModes:\n - ReadWriteOnce\n capacity:\n storage: 5Gi\n phase: Bound\n\n2. Wait for the updated capacity to reflect in PVC status\n\n kind: PersistentVolumeClaim\n metadata:\n name: jiva-csi-demo\n spec:\n resources:\n requests:\n storage: 10Gi\n volumeName: pvc-136b015f-dac3-11e9-8980-42010a80006c\n status:\n accessModes:\n - ReadWriteOnce\n capacity:\n storage: 10Gi\n phase: Bound\n\n3. Verify the updated size of the mounted volume inside the application pod\n\n```\n# df -h /storage\nFilesystem \tSize Used Available Use% Mounted on\n/dev/sdb \t9.7G \t11.3M 9.7G 0% /storage\n```\n\n## Volume Scaleup\n\n**Notes:**\n\n* Only 1 replica at a time can be scaled up.\n* Scale down is not supported.\n\n**Steps:**\n\n1. Increase the desiredReplicationFactor in the jivaVolume spec section by 1. This will spawn another replica for the sts and the data will be rebuilt on the new replica.\n\n```\napiVersion: openebs.io/v1alpha1\nkind: JivaVolume\nmetadata:\nannotations:\n openebs.io/volume-policy: example-jivavolumepolicy\ncreationTimestamp: \"2021-03-22T12:24:23Z\"\ngeneration: 23\nlabels:\n nodeID: k8s-worker-1\n openebs.io/component: jiva-volume\n openebs.io/persistent-volume: pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nname: pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nnamespace: openebs\nresourceVersion: \"170894664\"\nselfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/jivavolumes/pvc-26dc4d24-1e2e-4727-9804-bcd7ce40364d\nuid: ed2ce0b4-acab-48e8-b659-8753be96f345\nspec:\naccessType: mount\ncapacity: 4Gi\ndesiredReplicationFactor: 3\n......\n```\n\n\n\n","notHasFeatureImage":true,"slug":"provisioning-openebs-jiva-volumes-via-csi"},{"id":8,"title":"Deploying YugabyteDB on Google Kubernetes Engine with OpenEBS","author":"OpenEBS","author_info":"OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes","date":"05-04-2021","tags":["OpenEBS","Open Source","Yugabyte","Cloud Native Gke"],"excerpt":"In this blog post, we’ll walk you through the necessary steps to get a 3 node YugabyteDB cluster running on top of GKE, backed by OpenEBS.","content":"\n[OpenEBS](https://www.openebs.io/) is a CNCF project backed by [MayaData](https://mayadata.io/) that provides cloud-native, open source container attached storage (CAS). OpenEBS delivers persistent block storage and other capabilities such as integrated back-up, management of local and cloud disks, and more. For enterprise cloud-native applications, OpenEBS provides storage functionality that is idiomatic with cloud-native development environments, with granular storage policies and isolation that enable cloud developers and architects to optimize storage for specific workloads.\n\nBecause [YugabyteDB](https://www.yugabyte.com/) is a cloud-native, distributed SQL database that runs in Kubernetes environments, it can interoperate with OpenEBS and many other CNCF projects.\n\n**Wait, what is YugabyteDB?** _It is an open source, and high-performance distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner. Yugabyte’s YSQL API is PostgreSQL wire compatible._\n\nIn this blog post we’ll walk you through the necessary steps to get a 3 node YugabyteDB cluster running on top of GKE, backed by OpenEBS.\n\n[![Deploying YugabyteDB on Google Kubernetes Engine with OpenEBS](/images/blog/deploying-yugabytedb-on-google-kubernetes-engine-with-openebs-video-preview.png)](https://player.vimeo.com/video/530995643?app_id=122963)\n\n\n**Why OpenEBS and YugabyteDB?** \nBecause YugabyteDB is a transactional database often used as a system of record, it needs to be deployed as a StatefulSet on Kubernetes and requires persistent storage. OpenEBS can be used for backing YugabyteDB local disks, allowing the provisioning of large-scale persistent volumes. \n\nHere are a few of the advantages of using OpenEBS in conjunction with a YugabyteDB database cluster:\n\n- There’s no need to manage the local disks as OpenEBS manages them.\n- OpenEBS and YugabyteDB can provision large size persistent volumes.\n- With OpenEBS persistent volumes, capacity can be thin provisioned, and disks can be added to OpenEBS on the fly without disruption of service. When this capability is combined with YugabyteDB, which already supports multi-TB data density per node, this can prove to be massive cost savings on storage.\n- Both OpenEBS and YugabyteDB support multi-cloud deployments helping organizations avoid cloud lock-in.\n- Both OpenEBS and YugabyteDB integrate with another CNCF project, [Prometheus](https://prometheus.io/). This makes it easy to [monitor both storage and the database](/docs/introduction/features#prometheus-metrics-for-workload-tuning) from a single system.\n\nAdditionally, OpenEBS can do [synchronous replication](/docs/introduction/features#synchronous-replication) inside a geographic region. In a scenario where YugabyteDB is deployed across regions, and a node in any one region fails, YugaByteDB would have to rebuild this node with data from another region. This would incur cross-region traffic, which is more expensive and lower in performance. But, with OpenEBS, this rebuilding of a node can be done seamlessly because OpenEBS is replicating locally inside the region. This means YugabyteDB does not end up having to copy data from another region, which ends up being less expensive and higher in performance. In this deployment setup, only if the entire region failed, YugabyteDB would need to do a cross-region node rebuild. Additional detailed descriptions of OpenEBS enabled use cases can be found [here](/docs/introduction/usecases).\n\nOk, let’s get started!\n\n**Prerequisites** \n\n![Yugabyte work flow](/images/blog/yugabyte-work-flow.png)\n\nUsing the latest and greatest versions of the available software (as of this blog’s writing), below is the environment which we’ll use to run a YugabyteDB cluster on top of a Google Kubernetes Engine (GKE) cluster backed by OpenEBS\n\n1. YugabyteDB - [Version 2.5.3.1](https://docs.yugabyte.com/latest/quick-start/install/)\n2. OpenEBS - [Version 2.7.0](https://github.com/openebs/openebs)\n3. A [Google Cloud Platform](https://cloud.google.com/gcp/) account\n\n**Step 1: Setting Up a Cluster on GKE** \nTo deploy YugabyteDB on the Google Cloud Platform (GCP), we first have to set up a cluster using Ubuntu as our base node image.\n\n**Note**: _GKE’s Container-Optimized OS does not come with an iSCSI client pre-installed and does not allow the installation of an iSCSI client. Therefore, we’ll be using the Ubuntu with Docker image type for our nodes._\n\nFor the purposes of this demo, I used the Google Cloud Console to configure my Kubernetes cluster. Aside from the typical defaults, here’s the options under the* Node Pools > default-pool > Nodes* I selected\n\n- **Image Type:** Ubuntu with Docker\n- **Series:** N1\n- **Machine Type:** n1-standard-4 (4 vCPU, 15 GB memory)\n\n![Yugabyte nodes](/images/blog/yugabyte-nodes.png)\n\nClick _Create_ and wait for the Kubernetes cluster to come online.\n\n**Step 2: Configure iSCSI** \nThe iSCSI client is a prerequisite for provisioning cStor and Jiva volumes. However, it is recommended that the iSCSI client is setup and *iscsid* service is running on worker nodes before proceeding with the OpenEBS installation. In order to set up iSCSI, we’ll first need to determine the names of the nodes in our cluster\n\n $ kubectl get nodes\n\n NAME \tSTATUS ROLES \tAGE \tVERSION\n gke-cluster-1-default-pool-be95f6dd-5x65 \tReady \t18h \tv1.18.15-gke.1501\n gke-cluster-1-default-pool-be95f6dd-rs6c \tReady \t18h \tv1.18.15-gke.1501\n gke-cluster-1-default-pool-be95f6dd-t4cp \tReady \t18h \tv1.18.15-gke.1501\n\n Now that we have the names of our nodes, we’ll want to log into each node and enable the iSCSI service.\n\n $ gcloud compute ssh \n $ sudo systemctl enable iscsid && sudo systemctl start iscsid\n\n You can check the status of the iSCSI service using the following command:\n\n $ systemctl status iscsid\n\n iscsid.service - iSCSI initiator daemon (iscsid)\n Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)\n Active: active (running) since Fri 2021-03-26 02:25:42 UTC; 18h ago\n Docs: man:iscsid(8)\n Process: 10052 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)\n Process: 10038 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)\n Main PID: 10059 (iscsid)\n Tasks: 2 (limit: 4915)\n CGroup: /system.slice/iscsid.service\n ├─10057 /sbin/iscsid\n └─10059 /sbin/iscsid\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 systemd[1]: Starting iSCSI initiator daemon (iscsid)...\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 iscsid[10052]: iSCSI logger with pid=10057 started!\n Mar 26 02:25:42 gke-cluster-1-default-pool-be95f6dd-5x65 systemd[1]: Started iSCSI initiator daemon (iscsid).\n\n**Step 3: Install OpenEBS** \nNext, let’s install OpenEBS. I’ve found that the OpenEBS Operator is one of the simplest ways to get the software up and running.\n\n $ kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml\n\nOnce the installation is completed, check and verify the status of the pods. You should something similar to this:\n\n $ kubectl get pods -n openebs\n\n NAME READY STATUS\n maya-apiserver-dd655ff87-rbgmd 1/1 Running\n openebs-admission-server-5965c94767-4h8rc 1/1 Running\n openebs-localpv-provisioner-5495669c66-z46lr 1/1 Running\n openebs-ndm-dss64 1/1 Running\n openebs-ndm-gnv75 1/1 Running\n openebs-ndm-operator-68949644b9-mqvlx 1/1 Running\n openebs-ndm-r5pws 1/1 Running\n openebs-provisioner-544cb85449-w9spl 1/1 Running\n openebs-snapshot-operator-6d65b778dd-79zcn 2/2 Running\n\n**Step 4: Create and Attach Disks to Nodes** \nOur worker nodes need to have disks attached. These disks need to be unmounted and not have a filesystem on them. To accomplish this we’ll need to execute the following commands on each node.\n\n $ gcloud compute disks create disk1 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-5x65 --disk disk1\n\n $ gcloud compute disks create disk2 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-rs6c --disk disk2\n\n $ gcloud compute disks create disk3 --size=10GB\n $ gcloud compute instances attach-disk gke-cluster-1-default-pool-be95f6dd-t4cp --disk disk3\n\n Next let’s verify that our block devices are indeed attached.\n\n $ kubectl get blockdevice -n openebs\n\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-03... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n blockdevice-85... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n blockdevice-b0... gke-cluster-1-default-pool-be9... 10736352768 Claimed Active\n\n**Step 5: Create a Storage Pool Claim** \nNow that we have the names of our block devices and have verified that they are active, the next step is to create a Storage Pool Claim. We’ll use this to then create a Storage Class, and finally use that for our Persistent Volume Claims. The first step in this chain of steps is to configure our Storage Pool Claim YAML file. In this demo, I’ve named it “cstor-pool1-config.yaml”.\n\n $ vim cstor-pool1-config.yaml\n\n #Use the following YAMLs to create a cStor Storage Pool.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-disk-pool\n type: disk\n poolSpec:\n poolType: striped\n blockDevices:\n blockDeviceList:\n - blockdevice-03e93d010db5169322eb16f3e18e33ed\n - blockdevice-22591882979084d0fe580fe229e0d84f\n - blockdevice-4d1b4bacbeec1650b337c2cfda7e3a48\n ---\n\n Once you’ve figured out how to exit vim, the next step is to create the resource.\n $ kubectl create -f cstor-pool1-config.yaml\n\nWe can verify our storage pool with the following command:\n\n $ kubectl get csp\n\n NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE\n cstor-disk-pool-6cmf 1.85M 9.94G 9.94G Healthy false striped\n cstor-disk-pool-jql6 40.6M 9.90G 9.94G Healthy false striped\n cstor-disk-pool-vbz5 68.2M 9.87G 9.94G Healthy false striped\n\n**Step 6: Create a Storage Class** \nNow that we have a storage pool, let’s configure the YAML file for our storage class. In this demo, I’ve named it “openebs-sc-rep1.yaml”.\n\n $ vim openebs-sc-rep1.yaml\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-sc-rep1\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk-pool\"\n - name: ReplicaCount\n value: \"1\"\n provisioner: openebs.io/provisioner-iscsi\n\nAssuming you have remembered how to exit vim from the previous step, we now need to create the storage class.\n\n $ kubectl create -f openebs-sc-rep1.yaml\n\nFinally, let’s verify the storage class.\n\n $ kubectl get sc\n\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE\n openebs-device openebs.io/local Delete WaitForFirstConsumer\n openebs-hostpath openebs.io/local Delete WaitForFirstConsumer\n openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate\n openebs-sc-rep1 openebs.io/provisioner-iscsi Delete Immediate\n openebs-snapshot... volumesnapshot.external... Delete Immediate\n premium-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer\n standard (default) kubernetes.io/gce-pd Delete Immediate\n standard-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer\n\nAt this point, we are now set up for Persistent Volume Claims.\n\n**Step 7: Install YugabyteDB** \nIn this final step we’ll install a 3 node YugabyteDB cluster running on top of GKE that will be backed by the OpenEBS deployment we just completed.\n\nThe first step is to create a namespace.\n\n_$ kubectl create namespace yb-demo_\n\nNext, let’s install the cluster using Helm.\n\n $ helm install yb-demo yugabytedb/yugabyte --set resource.master.requests.cpu=1,resource.master.requests.memory=1Gi,\\\n resource.tserver.requests.cpu=1,resource.tserver.requests.memory=1Gi,\\\n enableLoadBalancer=True --namespace yb-demo --set storage.master.storageClass=openebs-sc-rep1,storage.tserver.storageClass=openebs-sc-rep1 --set persistence.storageClass=openebs-cstor-disk --wait\n\nNote that in the command above we are specifying the following so that YugabyteDB makes explicit use of OpenEBS:\n\n- _storage.master.storageClass=openebs-sc-rep1_\n- _storage.tserver.storageClass=openebs-sc-rep1_\n- _persistence.storageClass=openebs-cstor-disk_\n\nOnce the installation is complete you should be able log into the PostgreSQL compatible YSQL shell on port 5433 with the following command:\n\n $ kubectl --namespace yb-demo exec -it yb-tserver-0 -- sh -c \"cd /home/yugabyte && ysqlsh -h yb-tserver-0\"\n\n ysqlsh (11.2-YB-2.5.3.1-b0)\n Type \"help\" for help.\n yugabyte=#\n\nYou can also access the basic YugabyteDB web admin portal at:\n\n_http://:7000_\n\n![Yugabyte master](/images/blog/yugabyte-master.png)\n\n**Viewing Services and Ingress** \nA quick and visual way to check out all the services and ingress is to go to the “Services and Ingress” view in the Google Cloud Console. If you’ve made it this far you should see something like this:\n\n![Yugabyte ingress](/images/blog/yugabyte-ingress.png)\n\nNote: I have omitted the “Endpoints” column from the screenshot above, but in your view you’ll be able to see the IPs and ports of the various endpoints.\n\nThat’s it! You now have a 3 node YugabyteDB cluster running on GKE with OpenEBS storage.\n\n**Next Steps** \nAs mentioned, MayData is the chief sponsor of the OpenEBS project. It offers an enterprise-grade OpenEBS platform that makes it easier to run stateful applications on Kubernetes by helping get your workloads provisioned, backed-up, monitored, logged, managed, tested, and even migrated across clusters and clouds. You can learn more about MayaData [here.](https://mayadata.io/)\n\n- Learn more about OpenEBS by visiting the [GitHub](https://github.com/openebs/openebs) and [official Docs](/docs) pages.\n- Learn more about YugabyteDB by visiting the [GitHub](https://github.com/yugabyte/yugabyte-db) and [official Docs](https://docs.yugabyte.com/) pages.\n\n**About the author:**\n\n![Jimmy Guerrero](/images/blog/authors/jimmy-guerrero.png)\n\nJimmy Guerrero, VP Marketing, and Community at YugaByte.","notHasFeatureImage":false,"slug":"deploying-yugabytedb-on-google-kubernetes-engine-with-openebs"},{"id":9,"title":"Repeatable OpenEBS Mayastor deployments and benchmarks","author":"OpenEBS","author_info":"OpenEBS is the most widely deployed and easy to use open source storage solution for Kubernetes","date":"22-03-2021","tags":["Mayastor","OpenEBS"],"excerpt":"Learn about Repeatable OpenEBS Mayastor deployments and benchmarks","content":"\n## Introduction\n\nOpenEBS is one of the most popular Storage-related projects in CNCF, and the newest addition to OpenEBS - Mayastor, is a missing piece that has been absent from the Kubernetes stack for a long time - Kubernetes-native, high performance, distributed Software-Defined Storage or what is increasingly called Container Attached Storage (CAS).\n\nAs the lead developers of OpenEBS Mayastor, we want to be sure our message of an extremely high performing CAS is not only exciting, but also honest and easy to check. We want every interested user to be able to quickly and easily bring OpenEBS Mayastor up, properly tuned and ready for testing with whatever workload the user prefers to try.\n\nIn order to deliver on that promise, we have started a [“Demo Playground” project, open sourced on Github](https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground). Contributions and feedback are welcome.\n\n\n## OpenEBS\n\nOpenEBS is a project with multiple storage engines, with each engine providing the user with different feature sets as well as different usage and performance characteristics. The currently available options can roughly be split into two categories:\n\n* LocalPV: Excellent for workloads that deal with storage resilience at the application level, creating and managing their own replicas and capable of sustaining the loss of a single or multiple nodes, such as Cassandra, and requiring very good storage performance, especially latency-wise.\n* Replicated storage (cStor, Jiva) - for workloads that are less performance-sensitive and some of the more advanced storage features such as synchronous data replication, snapshots, clones, thin provisioning of data, high resiliency of data, data consistency, and on-demand increase of capacity or performance.\n\nAdvanced features come at the cost of higher latency and lower performance, and yet, technology keeps advancing and trying to get the best of both worlds.\n\n\n## OpenEBS Mayastor\n\nOpenEBS Mayastor delivers on the promise of exciting new technology, utilizing NVMe (not just the disks, but the protocol and standards), NVMEoF, SPDK and io_uring. NVMes inside our servers deliver amazing speeds and latencies, huge numbers of IOPS, and using old SCSI or FC protocols only waste resources introducing overheads. Harnessing SPDK and NVMEoF OpenEBS Mayastor achieves speeds that are close to in-host NVMes, without compromising on workload mobility, resilience, flexibility, and enterprise features.\n\nStill, all this exciting tech needs some proper care before it behaves as it should, and we still have a ways to go before it autotunes and autoconfigures itself just right with the help of Kubernetes and workload operators; and yet, as a user willing to take Mayastor for a spin, there should be no reason to wait, if the tuning and preparation can be automated now.\n\n\n## Introducing: the Automation Playground\n\nThe Automation Playground provides an easy onramp for trying out OpenEBS Mayastor in a cloud or self-hosted environment and attempts to keep the installation process correct, standardized, and consistently reproducible, yet both simple and flexible.\n\nThe Playground utilizes popular and familiar software in order to apply the desired state configuration, as well as following a familiar Jenkins-pipeline-like approach.\n\nThe entire process is split into stages, with each stage extensible, replaceable and skippable, if need be, and each stage is called from a simple bash script, where each step is a function, easily copied into a CI engine as a pipeline stage.\n\nThe user experience is as simple as editing a single variables file in order to define the benchmark setup variables and running up.sh. The script will then iterate over the predefined stages, relying on the outcomes of each stage to run the next one\n\nVariables are used to define such things as the setup name (prefixed in all the provisioned resources), user access credentials, Kubernetes installation types, provisioning details, and of course, OpenEBS Mayastor tuning as well as the benchmark itself. For more details, please see the vars file at [https://github.com/mayadata-io/deployment-automation-playground/blob/main/demo-playground/vars](https://github.com/mayadata-io/deployment-automation-playground/blob/main/demo-playground/vars)\n\n\n## Stages\n\nEach software lifecycle consists of several stages - provisioning, deployment, operations, and teardown.\n\nSince we are flexible here, each stage can be skipped if it isn’t required in a given setup.\n\nWhen running a benchmark on a set of self-hosted bare metal machines, the provisioning stage is not needed.\n\nIf Kubernetes is already installed, the Kubernetes installation stage can be skipped.\n\nWhen running the Demo Playground on a host that has direct access to the machines executing the benchmark, the VPN stage can be skipped.\n\nThe only truly essential stages are node preparation and the actual OpenEBS Mayastor workload playbooks that will be installed.\n\n\n#### Stage 1: Provisioning\n\nAt this step, we use Terraform to create a separate environment for the benchmark. Currently, the supported provisioning options are Azure and AWS EC2, with GCP support not too far behind. As a reminder, contributions (and feedback) are welcome.\n\nTerraform is used to create a separate VPC (in EC2) or Resource Group (in Azure), where networking is configured, and VMs are provisioned as per the definitions in the vars file.\n\nThe nodes provisioned are of three varieties\n\n* Master nodes (for Kubernetes Masters)\n* Worker nodes (Kubernetes workers that will be running the workload - make sure these are powerful enough and include fast networking if you want to be able to stress Mayastor)\n* Storage nodes (Kubernetes workers that will be running Mayastor). These instances should have fast local NVMe disks, which means LXs_v2 on Azure, m5d/m5ad/m5dn/i3 on AWS or n1/n2_standard with added Local-SSDs on GCP.\n\nWhen provisioning is complete, an Ansible inventory file is generated by Terraform, to be used in later stages. The inventory contains all the node IPs split into groups and adjusted for the various Kubernetes installers in use.\n\nIf the provisioning stage is skipped, the user must provide the inventory.ini file in the workspace directory, with the file containing the [mayastor_clients] (non-storage workers) and [mayastor_storage] (storage nodes) groups.\n\n#### Stage 2: Start VPN\n\nThis is a small stage, only required when the host executing Demo Playground is not inside the same subnet as the cluster nodes. The stage starts sshuttle after creating a script in the workspace directory. Sshuttle is described as a “poor man’s VPN” - an easy to use package that will tunnel all traffic for a given subnet through an SSH tunnel to a Bastion host.\n\nDuring provisioning, the first Kubernetes Master host has designated the Bastion and will be used for this purpose, effectively working as a VPN concentrator for the Demo Playground setup, placing the executor host in the same subnet as the Kubernetes nodes.\n\n#### Stage 3: Kubernetes setup\n\nAt this step, the Playground will deploy a pre-configured version of Kubernetes on the hosts as described in the inventory. If Provisioning was skipped, this means that the inventory file will have to be expanded with groups that are pertinent to the Kubernetes deployment in use; otherwise, the inventory generated in the Provisioning stage will contain all the required groups.\n\nCurrently two installation types are supported with more planned:\n\n* Kubespray - a well known Ansible based feature rich Kubernetes installer\n* K3S - a simplified and downsized Kubernetes distribution, which can be perfect for a small demo setup. This is also installed via Ansible.\n\nAt the end of the step, the script will extract the KUBECONFIG credentials file from a Master node and place it under workspace/admin.conf. If this stage is skipped, the user will have to extract and add this file manually.\n\n#### Stage 4: Node preparation\n\nIn order to run OpenEBS Mayastor as well as other OpenEBS storage engines, some prerequisites need to be applied to the Kubernetes workers, both the storage and client nodes.\n\nThis includes making sure the iSCSI and NVMeo-TCP client packages are present, installing and enabling the various Linux kernel modules, enabling hugepages, and so on. Some of these settings might require a host restart.\n\nThe stage is implemented as an Ansible playbook, which allows it to reach into the hosts directly in order to prepare them, performing some actions a Kubernetes pod has limited access to.\n\nAt this point, we should have a working Kubernetes setup, with the different worker nodes prepared for using Mayastor either as storage hosts or storage clients.\n\n## Playbooks\n\nActually, the proper stages end at Node Preparation, and then the playbooks take over. The vars file contains a PLAYBOOKS variable, which lists all the playbooks the Playground will apply in sequence.\n\nCurrently, there is one playbook relevant to testing Mayastor - mayastor.yml\n\nBut the script will attempt to run any playbooks mentioned from the deployments directory one after another.\n\nThe Mayastor playbook follows the Mayastor installation instructions, creating the Kubernetes manifests and applying them to the setup, so that all the relevant Mayastor pods, DaemonSets, StorageClasses, Pools etc. are created in the Mayastor namespace, PVCs are created and ready to be used by the user’s workload.\n\nThe Mayastor playbook also contains an optional FIO test, which will create an FIO pod using the first created PVC and run a quick 1-minute benchmark.\n\n## Conclusion\n\nThe Demo Playground project is still in very early stages, and we invite everyone to use, contribute and expand upon it. The goal here is to give the user interested in giving OpenEBS Mayastor a try, a ready tool that does the job in an open, honest, consistent, and reproducible manner.\n\nThe project’s flexibility allows for anyone to add in additional playbooks, which will deploy and run different workloads on top of Mayastor, and we intend to expand upon it, adding some workloads of our own beyond the basic FIO benchmark.\n\nPlease visit us at [https://mayadata.io](https://mayadata.io) and give the Demo Playground a spin at [https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground](https://github.com/mayadata-io/deployment-automation-playground/tree/main/demo-playground).\n\nYou can also find my colleagues and me spending time on the Kubernetes #OpenEBS slack, or at a [Discord room](https://discord.com/invite/zsFfszM8J2) set up to focus mostly on open source collaboration with Mayastor developers (Rusticians may be especially interested), and on the Data on Kubernetes community where a huge variety of users of Kubernetes for data are sharing their perspectives (https://dok.community/).\n\n\n****About the author:****\n\n![Dan Yasny](/images/blog/authors/dan-yasny.png)\n\nDan Yasny is a Principal Field Engineer at MayaData, previously he worked as a Field Engineer at ScyllaDB, an SDET, Technical Product Manager and a Sustaining Engineer at Red Hat, working on such projects as ScyllaDB, Kubernetes, OpenShift, KubeVirt, OpenStack, oVirt/RHV and more.","notHasFeatureImage":false,"slug":"repeatable-openebs-mayastor-deployments-and-benchmarks"},{"id":10,"title":"How are TikTok, Flipkart, KubeSphere, and others using OpenEBS for Local Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"12-03-2021","tags":["LocalPV","OpenEBS","Flipkart","TikTok","Kubernetes","Mayastor","MayaData"],"excerpt":"How are TikTok, Flipkart, KubeSphere, and others using OpenEBS for Local Volumes","content":"\n**How to select the right local volume for your workloads?**\n\nWe have recently seen a massive increase in the usage of different flavors of OpenEBS Local PV. We estimate by looking at container pulls for underlying components combined with some call home data for those users of OpenEBS that enable the capturing of metrics that the weekly new deployments of OpenEBS for LocalPV increased by nearly 10x during 2020. This can be attributed to the fact that more and more cloud native Stateful applications are moving into Kubernetes\n\n![LocalPv Deployment](/images/blog/local-pv-deployment.png)\n\nSome of the prominent users of OpenEBS Local PV include the CNCF, Optoro, ByteDance / TikTok, Flipkart, and many more. You can always read more about OpenEBS users on the OpenEBS.io website and on the GitHub project page here: [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md).\n\nWhile Kubernetes provides native support or interfaces for consuming Local Volumes, the adoption of OpenEBS for LocalPV management suggests that some capabilities are missing that are desired by users. At a high level, dynamic provisioning and the simplicity of deleting Local Volumes are two reasons often given for the preference of some users for the use of OpenEBS LocalPV.\n\nIn this blog, I outline the various types of Local Storage that users have in their Kubernetes clusters and introduce the various flavors of OpenEBS Local PV being used.\n\nBefore getting into the flavors of OpenEBS Local PV, it might be worthwhile to know what Kubernetes offers or means by a Local Volume.\n\n_A [Kubernetes Local Volume](https://kubernetes.io/docs/concepts/storage/volumes/#local) implies that storage is available only from a single node. A local volume represents a mounted local storage device such as a disk, partition, or directory._\n\nSo, it stands to reason - as the Local Volume is accessible only from a single node, local volumes are subject to the availability of the underlying node. If the node becomes unhealthy, then the local volume will also become inaccessible, and a Pod using it will not be able to run.\n\nHence, Stateful Applications using local volumes must be able to tolerate this reduced availability, as well as potential data loss, depending on the durability characteristics of the underlying disk.\n\nAs it happens, many of the Cloud Native Workloads - are distributed in nature and are typically deployed as StatefulSets with multiple replicas. These can sustain the failure or reduced availability of a single replica. MinIO, Redis, PostgreSQL, Kafka, Cassandra, Elastic are just some examples that are deployed using Local Volumes. For these applications - performance and consistent low latency, and ease of management are more important than the resiliency of a node to failures.\n\nAs the large SaaS provider, [Optoro](https://github.com/openebs/openebs/blob/master/adopters/optoro/README.md) puts it: \n*The vast majority of applications are able to better handle failover and replication than a block level device. Instead of introducing another distributed system into an already complex environment, OpenEBS's localPVs allow us to leverage fast local storage. … OpenEBS has allowed us to not introduce a complicated distributed system into our platform. The adoption has been smooth and completely transparent to our end users.*\n\n## Limitations of Kubernetes LocalPV\n\nKubernetes expects users to make Persistent Volumes (PVs) available that it can then associate with PVCs during scheduling. Kubernetes does not help with dynamically creating these PVs as the applications are launched into the cluster.\n\nThis pre-provisioning can become an issue when companies have more than two people or teams managing the Kubernetes clusters, and the Application teams depend on the Kubernetes cluster administrators for provisioning the Volumes.\n\nWe have seen that cluster administrators are challenged by the following aspects:\n\n(a) The type of storage available on the Kubernetes nodes varies depending on how the Kubernetes nodes are provisioned. Available storage types include:\n\n- Nodes have only OS disks with large space that can be used for provisioning Local Volumes.\n- Nodes have one or two additional devices (SSDs or Disks) attached that can be used for provisioning Local Volumes.\n- Nodes have 8 to 16 high-performing NVMe SSDs.\n\n(b) And then, there is a matter of capacity available from the Local Storage and how to manage this to enable the freedom of developers and other consumers of capacity while retaining a level of oversight and assistance by centralized teams:\n\n(c) First, the platform or other centralized team may not know exactly what the capacity a particular team or workload needs - and the developer or data scientist may not know either. Dynamic provisioning within quotas means that users can keep moving without opening a ticket or having a conversation.\n\n(d) Secondly, there are many common operations tasks that need to be performed. Just because the applications are resilient does not mean these tasks entirely disappear. Administrators still would like to safeguard the data with best practices from years of experience in dealing with data such as:\n\n- Enforcing Capacity Limits/Thresholds\n- Securing the Volumes\n- Carving out the Local Volumes from well known or familiar file systems like LVM, ZFS, XFS, and so forth\n- Encrypting the Volumes\n- Enforce compliance with BCP by taking regular snapshots and full backups\n\nThis is where Kubernetes itself stops, and plugins like OpenEBS LocalPV options step into the auto-magically provision and manage the Local Volumes.\n\n## Selecting your LocalPV\n\nOpenEBS provides different types of Local Volumes that can be used to provide locally mounted storage to Kubernetes stateful workloads. The choice of the OpenEBS Local Volume depends on the type of local storage available on the node and the features required.\n\n- OpenEBS Hostpath Local PV - The use of the host path is the simplest, most used, and lowest overhead solution. This approach creates Local PVs by creating a sub-directory per Persistent Volume. This offers flexibility to create different classes of storage and allows administrators to decide into which parent or mounted directory the Persistent Volumes sub-directories should be placed. For example - a storage class for critical workloads vs. non-critical transient workloads, SSD vs. Hard Disk mounted paths, and so forth.\n- OpenEBS Raw file Local PV - The OpenEBS Raw file approach evolved out of the Hostpath approach due to considerable feedback from some OpenEBS community members. Raw file Local PV offers all the benefits of Hostpath Local PV - and in addition, Hostpath supports enforcing Capacity Quotas on Volume subdirectories by creating sparse files per volume.\n- OpenEBS Device Local PV - Device Local PV is best suited for cases where either a complete device or a partitioned device needs to be dedicated to the pod. Workloads like Cassandra or Kafka that need high throughput and low latency often use dedicated device Local PV.\n- OpenEBS ZFS and LVM Local PV - Both ZFS and LVM are selected by seasoned storage administrators that want to leverage all the good things of well-known filesystems or volume management along with the power of Local Volumes. This category offers features like full/incremental snapshots, encryption, thin-provisioning, resiliency against local disk failures by using software raid/mirror, and so forth. Incidentally, you can easily cause a fairly reasoned argument by asking users and community members, and even our own engineers to share their opinions about whether ZFS or LVM is more useful; I'm very happy that the community has progressed to the point that both solutions are now supported and widely deployed.\n\nI hope this overview of LocalPV options and OpenEBS Local has been useful. I plan to follow this with further blogs that get into the details of each flavor of the OpenEBS Local PV.\n\nIn the meantime, you can get started easily with [OpenEBS Local PV](/docs), and the community is always available on the Kubernetes Slack #openebs channel.\n\nOr read more on what our OpenEBS users and partners have to say about Local PV. From our friends at 2nd Quadrant (now part of EDB): [Local Persistent Volumes and PostgreSQL usage in Kubernetes](https://www.2ndquadrant.com/en/blog/local-persistent-volumes-and-postgresql-usage-in-kubernetes/)\n\nAnd from one of the most broadly deployed Kubernetes distributions, Kubesphere: [OpenEBS Local PV is default Storage Class in Kubesphere](https://github.com/openebs/openebs/tree/master/adopters/kubesphere)\n\nOr, again, you can find more stories and can add your own to Adopters.MD on the OpenEBS GitHub: [https://github.com/openebs/openebs/blob/master/ADOPTERS.md](https://github.com/openebs/openebs/blob/master/ADOPTERS.md)\n","notHasFeatureImage":false,"slug":"how-are-tiktok,-flipkart,-kubesphere,-and-others-using-openebs-for-local-volumes"},{"id":11,"title":"OpenEBS NDM, go-to solution for managing Kubernetes Local Storage","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"13-01-2021","tags":["OpenEBS"],"excerpt":"Read about OpenEBS NDM, the go-to solution for managing Kubernetes Local Storage.","content":"\nEver since Local Volumes have become generally available (GA) in Kubernetes 1.14, the use of Local Volumes has skyrocketed. This can be attributed to the nature of cloud-native workloads distributed in nature and can sustain node failures. The bare metal underpinning Kubernetes clusters, both on-prem and cloud, can now be configured with local storage to manage stateful workloads. Kubernetes doesn’t treat storage like a native resource on par with CPU or Memory, making it a little difficult to make Kubernetes work out of the box to create effective node-attached storage. OpenEBS NDM helps alleviate this gap by discovering the different storage types attached to each worker node and then creating Kubernetes resources called block devices.\n\nApplication or storage operators can then use the information exposed via block devices to determine how to orchestrate the workloads best.\n\nOpenEBS NDM (Node Device Manager) has been declared GA after being deployed in production for several months as part of the OpenEBS control plane. With the release of version 1.0, NDM adds out-of-the-box support for partitions, LVMs, LUKS encrypted devices, in addition to the unique identification of virtual disks within the cluster. Now offering support for partitions, a single disk can be partitioned. Each partition will be considered a separate block device used by different storage engines like cStor / local PV. NDM also tracks the movement of the devices within a cluster across the nodes.\n\n## Key Storage Problems solved by NDM\n\n* Local Storage Discovery - detecting partitions, devices used as a LUKS device or LVM device, or if it can be accessed as a raw block device.\n* Cluster-wide storage visibility\n* Detect the movement of storage devices across nodes\n* Book-keeping/storage asset management - allocating/reserving, which type of storage should be provided to which workloads.\n\n## Getting Started with NDM\n\nLet us see how NDM helps detect the block devices in the cluster with 3 nodes, each having a completely different disk configuration. The Disk configuration of the nodes are as follows:\n\nMaster: 2 virtual disks\n\nWorker1: 3 virtual disks, one being used by LUKS and two other disks which are partitioned, several partitions are being used as PV's by the LVM.\n\n![](https://lh3.googleusercontent.com/7r1RKQF4udqvigbryA6XFOxRuoOccQSFqhM5C_e27ArTSXnsXIXZk7b3lwgJm4C2VxxWj4rHoED-pZl4PS_KVkF_SC4D2-NLJzokpg2cqlP2upSNva5PLCaBKtQCBueUhWFYTtS9)\n\n\nWorker 2: 4 physical disks\n\n* Deploy NDM into the Kubernetes cluster along with OpenEBS LocalPV\n ```\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-lite.yaml\n ```\n (The latest helm charts for deploying NDM are available [here](https://openebs.github.io/node-disk-manager/))\n\n* Once deployed, check the blockdevices present in the cluster using\n ```\n kubectl get bd -n openebs -o wide\n ```\n\n![](https://lh4.googleusercontent.com/v-iVUrfW6v3wSaXmb06pbek7as_RfFTlRJCmsPzhmId460JIsP0LvXVDBkA0FUnBdO3yt203HqHIBYorT-nP6ZtCZTKdRcao0Ws3tlNyvz8yQF9ytQN_UXxbyO9ZFs6-PeLYHQOD)\n\nSome block devices show partitions that did not exist initially. E.g., sdb1 instead of sdb. This is because NDM creates a partition on virtual disks to identify the disk uniquely. Also, block device resources are now created for LVMs and LUKS encrypted devices. All the block devices listed above will now be treated as individual devices and can be used by any storage engine.\n\n* Deploy a sample application to use the block device\n\nDownload the minio yaml and apply it. (NOTE: A node selector has been added to the minio application pod so that it gets scheduled on worker-1)\n```\nkubectl apply -f [minio-official.yaml](https://gist.githubusercontent.com/akhilerm/194a1606c514d8930addcaef56f9f19f/raw/7d339e5042b4e5e958dde558f1f3509e26c214f3/minio-official.yaml)\n```\nNow check the status of block devices again \n\n![](https://lh3.googleusercontent.com/A_JL0jXsZhmIPPrRYCSeMHVcPsey6ahFYV1_LVUapmbPLTrcgGEAao_ohbx9zU_SZl-lHmKGYgdMqh4czUCISSezbcOi4rznQNuX0sTAomO4y5HQLVYicTD4s1mPOZfUciacEOU_)\n\nWe can see that the device `dm-2`, is the LUKS device, has been claimed and used by the application.\n\n* Pool movement across nodes\n\n NDM helps in seamlessly moving cStor pools from one node to another. Whenever the devices that constitute a pool are moved from one node to another (disconnecting disks from one node and reconnecting on another), the block device resource is updated with the latest information. NDM tracks this movement. cStor can use this information to migrate pools as required.\n\n* Reserving storage for workloads\n\n NDM provides a feature to reserve devices for certain workloads. E.g., Users can reserve all SSDs for a performance intensive workload. This reservation is achieved using block-device-tags. More information on using block-device-tags with LocalPV can be found [here](/docs/user-guides/localpv-device#optional-block-device-tagging).\n\n## Future Roadmap\n\n* Southbound provisioning\n* Metrics (currently in alpha)\n* API Service to interact with NDM\n* Ability to create partitions or LVM volume groups - preparing storage in general.\n\n## Interested in Contributing to NDM?\n\nNDM is an OpenEBS project, which itself is a CNCF sandbox project. [OpenEBS on GitHub](https://github.com/openebs/node-disk-manager) is a great place to join if you want to contribute to our codebase. You can also interact with us on the OpenEBS channel in [Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F).","notHasFeatureImage":false,"slug":"openebs-ndm,-go-to-solution-for-managing-kubernetes-local-storage"},{"id":12,"title":"Storage is Evolving!","author":"Nick Connolly","author_info":"Nick is the Chief Scientist at MayaData and a pioneer of storage virtualization, holding patents ranging from highly-scalable algorithms through to data protection techniques.","date":"11-12-2020","tags":["OpenEBS"],"excerpt":"Learn how storage has evolved over the years. ","content":"\nBefore the turn of the century, storage systems were typically controlled by dedicated firmware running on custom hardware. These proprietary systems were time-consuming to design, expensive to build, and resistant to innovation.\n\nIn 1998, Software-Defined Storage was pioneered by DataCore Software with its SANsymphony suite of products, based on the realization that general-purpose computers had become fast enough to handle the demands of a high-performance storage stack. For context, this was an era when a system with more than two cores was a rarity and both memory and storage were measured in MBs! The primary protocol in use in the enterprise was SCSI, whether directly connected or accessed through a Fibre Channel network, response times were measured in the tens of milliseconds, and accessing storage over Ethernet using iSCSI was only just starting to be worked on.\n\n## The hardware environment is changing!\n\nIn the last few years, the hardware environment has changed significantly. Instead of the relentless drive for ever-increasing clock speeds, systems with over a hundred cores are now mainstream. Developing highly-performant algorithms that operate at this scale of parallelism is a complex and time-consuming process that, generally speaking, is uneconomic to pursue. Storage media has also undergone a transformation, with SSDs based on flash memory delivering orders of magnitude better performance than spinning disks. Their response time, which can be measured in microseconds, has highlighted the inefficiencies of the decades-old SCSI protocol.\n\nNVMe is a ‘state of the art’ storage protocol for a new era. Designed from the ground up for maximum parallelism and lock-free operation, it offers up to 64k independent I/O queues each with 64k entries and a simplified command set. Connected over PCIe, it delivers low latency and high bandwidth data directly to an application, enabling it to fully utilize the capabilities of the underlying flash memory. NVMe over Fabrics (NVMe-oF) provides network access to remote storage and targets less than 10 microseconds in additional latency.\n\n## Application development is changing!\n\nRather than building the large monolithic codebases that were the norm at the turn of the century, modern development practices are based around composable architectures; containerized microservices that scale dynamically to meet performance requirements. For more background on this trend, see my [earlier post](https://www.datacore.com/blog/5-changes-that-are-reshaping-software-development/) and the excellent articles in [MayaData’s blog](https://blog.mayadata.io/). Kubernetes is rapidly becoming the control plane for the enterprise.\n\n## A New Era\n\n![New Era](https://lh3.googleusercontent.com/5C8pUrteH4V8JB1li4myidOdIP1xAefDES3ksqG1SaxFX4YHhFZz2gX-tNQV7n4UVuHS-BvZejBVnDnLJiwte6LgGgHN2dzsKDKxC2cd-popha9Ljnw9CWNQ2JUvL_1a2F-w8x0i)\n\nA new era requires a new kind of storage stack! A stack that is based around today’s technologies rather than being anchored to the last century. A stack that is portable and flexible. A stack that supports rapid innovation. That delivers the performance that applications require.\n\n## Container Attached Storage\n\nThe new category of [Container Attached Storage](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/), of which OpenEBS is the de-facto open source standard, orchestrates the storage stack with the same flexibility as the application. Implemented as a microservices based architecture, it runs within Kubernetes and gives users the freedom to define the way that they want to access, protect, and manage their data. The days of the dedicated storage administrator are coming to an end!\n\nFor Mayastor, the latest storage engine to be added to OpenEBS, flexibility, and performance are achieved by basing the stack around the [Storage Platform Development Kit (SPDK)](https://spdk.io/), which provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. Based on the NVMe protocol, it delivers blistering performance from today’s hardware as well as being ready for the next generation of Intel Optane based SSDs that are just becoming available. For more details, see some [recent results](https://openebs.io/blog/mayastor-nvme-of-tcp-performance/).\n\n## Microsoft Windows\n\nHowever, amid all the discussions about flexibility and portability, there is one small footnote that often goes unnoticed: ‘not *supported on Windows*’. It’s understandable, because most of the projects that are shaping this new era have their roots on Linux or FreeBSD, but it overlooks the sheer scale of Windows Server deployments in enterprise environments. Things are changing, with significant investments being made in Kubernetes on Windows, but it’s a slow process; one project at a time!\n\nMayaData’s mission is to enable data agility - so we were uncomfortable with our high-performance Container Attached Storage solution, OpenEBS Mayastor, not being available on Windows platforms. With that in mind, we have created the [Windows Platform Development Kit (WPDK)](https://github.com/wpdk/wpdk) to act as a foundational layer to make it easier to port the SPDK to Windows. In addition, we are working with the SPDK community to make a few changes to the code base to support this. It is a testament to the quality of the excellent SPDK project that so few changes have been required so far.\n\nThe project also benefits from the work done by the DPDK on Windows community who has invested a significant amount of time porting the underlying [Data Plane Development Kit (DPDK)](https://www.dpdk.org/), a Linux Foundation project that consists of libraries to accelerate packet processing workloads running on a wide variety of CPU architectures.\n\n## Windows Platform Development Kit\n\n![Windows Platform Development Kit](https://lh4.googleusercontent.com/UDp5t-uCJeM6QlsMpoZCz-oxp2CyYDPS1BMhkdeaXn4asIPhdLzy0GLG74xdceDyWAa8bCrijsMLOZfrwKC7vQyQLNS-uGJbGLXyeDtBljMvMNDQphRtcfgMJ65mhZBTC7v6wFwg)\n\nThe MayaData developed and contributed Windows Platform Development Kit has currently reached ‘alpha’. Most of the required functionality is believed to be present, unit tested, and working correctly, but there are still areas that need further development.\n\nIt is possible to build the SPDK tree, run the associated unit tests, serve an iSCSI target on Windows, and mount it as a volume.\n\nIt is anticipated that this collaboration will deliver the following benefits to Windows users:\n\n1. Enable high-performance access to NVMe storage directly from applications.\n2. Native software-defined storage stacks, including OpenEBS Mayastor.\n3. Support for NVMe-oF adaptors from manufacturers such as Mellanox and Broadcom.\n\nThe Windows Platform Development Kit is open source, under a BSD-3 clause license. Community contributions are welcomed and needed! To get started please head to [https://wpdk.github.io](https://wpdk.github.io) or access the WPDK code and documentation on [GitHub](https://github.com/wpdk/wpdk).","notHasFeatureImage":false,"slug":"storage-is-evolving!"},{"id":13,"title":"OpenEBS on DigitalOcean Marketplace","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"3-12-2020","tags":["OpenEBS"],"excerpt":"Learn how to deploy OpenEBS on the DigitalOcean marketplace","content":"\nDeploying OpenEBS on DigitalOcean can directly be done from the console. DigitalOcean provides the feature to create a cluster with OpenEBS deployed on it already. To get started, follow the below-mentioned steps:\n\nWORKFLOW:\n\n![Workflow](https://lh3.googleusercontent.com/fQOb_mUG5ebZ4eu2eLCZw4WFIiG_LOUgk2xXj4tBsXokE1oMu5H4SQDcx1jgbpLYBBn4gVpeDOwgU_DhagUjyHi4_kFL3evGUjVTIfkY3Xdf6071c6XWO6AoJ5PruG5f1njtvaJm)\n\nSTEP 1: Getting started \nLogin to your [DigitalOcean](https://cloud.digitalocean.com/login) account.\n\nSTEP 2: Creation of cluster \nOnce you log in, you arrive at the dashboard, click on Marketplace under DISCOVER located on the left sidebar.\n\nNext, scroll down to find OpenEBS. On clicking, you will be redirected to a page where you will find the details about OpenEBS and the Create OpenEBS button on the right side.\n\nNext, you need to provide the necessary details such as Data Center region, cluster capacity, cluster name, etc. (It is advisable to provision 3 nodes with 4vCPUs and 8 GB memory to ensure that the resources are sufficient at all times.)\n\n![Creation of cluster](https://lh3.googleusercontent.com/zvcGjrkGegKSp-t29NQf5XPHzCf6LqRn-XFJuRsxNZBfopwNibKiUwBDo9KSFGTWub6tEqLnl_IWKtCTykql315aqUlTqa7U1ORYJ5H4OmIhdVeHArPmRELvKk94GFLIbui9LJTx)\n\nSTEP 3: Connecting your cluster\nCreation, resizing, and deletion can be carried out from UI, but you require command-line tools from your local machine or a remote management server to perform administrative tasks. The detailed steps to install the management tools and connect the cluster to your local machine can be found under the Overview section.\n\n![Connecting your cluster](https://lh4.googleusercontent.com/5ftx1DgzSOKXRQ_UdiMmakqRMdOMnyes7n0l7kT23t50dSloHosbqAgx7zH2hqaMhE77KIoKINERafGDCyPgZbvnGNQ27oIvpeNm7YqCjGv-6lx9aAgQMSoHiE4j8BrdYABdPg7K)\n\nTo verify, execute the following command:\n\n```\n$ kubectl get ns\n```\n\nOutput:\n```\nNAME STATUS AGE\ndefault Active 13m\nopenebs Active 13m\n```\nThe output must contain openebs ns in an Active state.\n\nNext, execute:\n\n```\n$ kubectl get pods -n openebs\n```\n\nOutput:\n```\nNAME READY STATUS RESTARTS AGE\nopenebs-admission-server-5c4d545647-r4vgr 1/1 Running 0 13m\nopenebs-apiserver-56f77c9965-xft68 1/1 Running 2 13m\nopenebs-localpv-provisioner-64c67b5b89-czv8b 1/1 Running 0 13m\nopenebs-ndm-5f6nt 1/1 Running 0 13m\nopenebs-ndm-74njq 1/1 Running 0 13m\nopenebs-ndm-operator-7bc769dcff-b45bc 1/1 Running 1 13m\nopenebs-ndm-spttv 1/1 Running 0 13m\nopenebs-provisioner-755f465f9d-fr67l 1/1 Running 0 13m\nopenebs-snapshot-operator-7988fc8646-zpd98 2/2 Running 0 13m\n```\nAll the pods must be in a running state.\n\nSTEP 4: Attaching BlockDevices \nTo attach BlockDevices to the created nodes, click on Volumes on the left sidebar and then click on the Add Volume button.\n\n![Adding Volume](https://lh3.googleusercontent.com/D96l0ASgsYCKEoenZQS7r-i_bdmLMlQ2PxcxYGqYLilrFospNmLmnVwfZAT-VYBHvSP31U70bgjdo0WhUbSuDfM0mU84s3-BopEd0vxuEHlZg64cnzIwO7LlLPc9RjhL5ResDD0-)\n\nNext, select the volume size ( provision at least 30 GB), select the node(droplet) to which it gets attached and provides a name, then click on the Create Volume button. Repeat these steps for each node (droplet).\n\n![Attaching BlockDevices](https://lh3.googleusercontent.com/i2IkMHV3CmVK8_fgiWMtiXhqlbkWGyCoCXaz4a0hXAR49WEuzXg6s7lbMEZFGr6oXLLFVAsLTfgJTELlrMKTE4mi5aNjDSMKMZn9XCMCGlWLpeUaaC4VsRg2xFPgw0tXLuG2T2uq)\n\nNOTE:\n\n*For cStor, choose Manually Mount and Format under Choose Configuration Options.*\n\n*For Jiva, choose Automatically Format and Mount under Choose Configuration Options.*\n\n*After the BlockDevices get attached for all the nodes, you can see an output similar to the below image.*\n\n![Output after the BlockDevices get attached for all the nodes](https://lh4.googleusercontent.com/i9KOccFfCGjP-Q3E2KpR0YOV3EXDfTin4RbNZbgo9A0PTSYRUj8E989KqnzHYXMigbfE0FZWK1_V0Jg_lAvZKN9iShkxLIyMFGmO9uVEYWhcosL-xUNc-VnrXpYcbu1VDKE-5zOT)\n\nNext, you have to provision OpenEBS volumes. Click [here](/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume) to know more.","notHasFeatureImage":false,"slug":"openebs-on-digitalocean-marketplace"},{"id":14,"title":"Atlassian Jira Deployment on OpenEBS","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"20-11-2020","tags":["OpenEBS"],"excerpt":"Learn how to deploy Atlassian Jira on OpenEBS in this short post.","content":"\n***Jira*** Software is part of a family of products designed to help teams of all types manage work. Originally, **Jira** was designed as a bug and issue tracker. But today, **Jira** has evolved into a powerful work management tool for all kinds of use cases, from requirements and test case management to agile software development.\n\n## Requirements\n\n#### Install OpenEBS\n\nIf OpenEBS is not installed in your K8s cluster, this can be done from [here](/docs/user-guides/installation). If OpenEBS is already installed, go to the next step.\n\n#### Configure cStor Pool\n\nIf cStor Pool is not configured in your OpenEBS cluster, this can be done from [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools). Sample YAML named **openebs-config.yaml** for configuring cStor Pool is provided:\n\n```\n#Use the following YAMLs to create a cStor Storage Pool.\n# and associated storage class.\napiVersion: openebs.io/v1alpha1\nkind: StoragePoolClaim\nmetadata:\n name: cstor-disk\nspec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get blockdevices -n openebs`\n #\n # `Block devices` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n# Replace the following with actual disk CRs from your cluster `kubectl get blockdevices -n openebs`\n# Uncomment the below lines after updating the actual disk names.\n blockDevices:\n blockDeviceList:\n# Replace the following with actual disk CRs from your cluster from `kubectl get blockdevices -n openebs`\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n# - blockdevice-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n\n---\n```\n\n#### Create Storage Class\n\nYou must configure a StorageClass to provision cStor volume on the cStor pool. In this solution, we are using a StorageClass to consume the cStor Pool, which is created using external disks attached to the Nodes. Since Jira is a deployment application, it requires three replications at the storage level. So cStor volume replicaCount is 3. Sample YAML named **openebs-sc-disk.yaml** to consume cStor pool with cStor volume replica count as 3 is provided:\n\n```\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n name: openebs-cstor-disk\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"3\" \nprovisioner: openebs.io/provisioner-iscsi\nreclaimPolicy: Delete\n```\n\n### Deployment of Jira\n\nSample Jira Yaml:\n\n```\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n labels:\n app: jira\n name: jira\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: jira\n name: jira\n spec:\n containers:\n - name: jira\n image: \"doriftoshoes/jira:7.3.6\"\n resources:\n requests:\n cpu: \"2\"\n memory: \"2G\"\n volumeMounts:\n - name: \"jira-home\"\n mountPath: /opt/jira-home\n volumes:\n - name: jira-home\n persistentVolumeClaim:\n claimName: demo-vol1-claim\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: jira\n name: jira\nspec:\n ports:\n - port: 8080\n targetPort: 8080\n selector:\n app: jira\n type: LoadBalancer\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: demo-vol1-claim\nspec:\n storageClassName: openebs-cstor-disk\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10G\n```\n\nNext, apply both the ***Jira deployment*** and service to your Kubernetes cluster.\n\n```\nkubectl apply -f jira.yaml\n```\n\n#### Verify Jira Pods:\n\n#### Run the following to get the status of Jira pods:\n\n```\nkubectl get pods\n```\n\nFollowing is an example output:\n\n```\nNAME READY STATUS RESTARTS AGE\njira-5xxxxxxxx-2xxxx 1/1 Running 0 1d12h\n```\n\nThat's it for today's blog. Thanks for reading. Please leave your questions or feedback, if any, in the comment section below.","notHasFeatureImage":false,"slug":"atlassian-jira-deployment-on-openebs"},{"id":15,"title":"Mayastor NVMe-oF TCP performance","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"19-11-2020","tags":["Mayastor"],"excerpt":"Overview of using Mayastor to try out some of the fastest NVMe devices currently available on the market and see how we perform on top of these devices within k8s, using the container attached storage approach for which OpenEBS is well known.","content":"\nFor a while now, we have been saying that OpenEBS Mayastor is “high” performance and community members have written [blogs](https://medium.com/volterra-io/kubernetes-storage-performance-comparison-v2-2020-updated-1c0b69f0dcf4) showing that the performance of OpenEBS Mayastor indeed is much better or on par with others even when running on relatively slow cloud volumes. However, is Mayastor high performance or just “as fast” as the other things out there? \n\nIt used to be the case that CPUs were much faster than the storage systems they served. With modern NVMe, this does not ***have*** to be the case anymore. NVMe is a ***protocol*** that can go fast but does not have to be fast. What this means is that you can use NVMe as your transport protocol for any block device, not just flash. Yes, this is what Mayastor does. It is really useful to keep in mind the distinction between NVMe as a protocol and NVMe devices - you don’t need to use them together but, when you do, additional performance can be unlocked.\n\nIn this blog, we will be using Mayastor to try out some of the fastest NVMe devices currently available on the market and see how we perform on top of these devices within k8s, using the container attached storage approach for which OpenEBS is well known. We will show what happens when you marry NVMe as a protocol (embedded within Mayastor) and fast NVMe devices from our friends at Intel.\n\nBefore we get started, you might wonder how we came to this point that a new project like OpenEBS Mayastor was able to deliver amongst the fastest storage available today. Richard Elling of Viking / Sanmina recently wrote an excellent blog on the trends in hardware design that puts NVMe and OpenEBS Mayastor into context:  [https://richardelling.substack.com/p/the-pendulum-swings-hard-towards](https://richardelling.substack.com/p/the-pendulum-swings-hard-towards)\n\n## Hardware setup\n\nLet’s get to it. We will be using three machines that will run kernel version 5.8. The hardware configuration of each host is as follows:\n\n- Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz\n- Intel Corporation NVMe Datacenter SSD [Optane]\n- Mellanox Technologies MT28908 Family [ConnectX-6]\n\n## Baseline performance\n\nTo understand the performance of the device we will be using throughout the test, we run the following Fio workload:\n\n [global]\n ioengine=linuxaio\n thread=1\n group_reporting=1\n direct=1\n norandommap=1\n bs=4k\n numjobs=8\n time_based=1\n ramp_time=0\n runtime=300\n iodepth=64\n \n \n [random-read-QD64]\n filename=/dev/nvme1n1\n rw=randread\n stonewall\n \n \n [random-write-QD64]\n filename=/dev/nvme1n1\n rw=randwrite\n stonewall\n \n \n [random-rw-QD64]\n filename=/dev/nvme1n1\n rw=randrw\n stonewall\n\n![Mayastor NVME](/images/blog/mayastor-nvme1.png)\n\nThese numbers are incredibly high and are provided by a ***single*** device. Note that the benchmark itself is rather synthetic in the sense that, in practice, no workload is 100% random.\n\n## High-level overview of the experiments\n\nMy approach in this benchmarking is very much OpenEBS Mayastor “the hard way”. If you want an easier to use solution that for example automates pool creation and device selection and so on - we call that offering Kubera Propel (also open source btw). You can learn more about Kubera Propel on the [MayaData.io](https://mayadata.io/) website.  \n\nOn two of the nodes, we create a pool (MSP CRD) which we use in the control plane to determine replica placement. To construct pools, we must have what we call a persistence layer. We support several ways to access this persistence layer. To select a particular scheme we use URIs. In this case we will be using today the pcie:/// scheme. This means that Mayastor will directly write into the NVMe devices listed above. The nice thing is that from the user perspective, things do not change that much. We simply replace disks: [‘/dev/nvme0n1’] with disks: [‘pcie:///80:00.0’]. Note that this persistence layer is used to store the replicas of the PVC. Once we have this layer up and running, we will create storage classes and select that we want to use nvmf (NVMe-oF) as the transport layer between the replicas, resulting in NVMe all the way through. \n\nAfter we have deployed mayastor we applied to following two storage classes:\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: nvmf\n parameters:\n repl: '1'\n protocol: 'nvmf'\n provisioner: io.openebs.csi-mayastor\n ---\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: nvmf-mirror\n parameters:\n repl: '2'\n protocol: 'nvmf'\n provisioner: io.openebs.csi-mayastor\n\nNote that `protocol: `nvmf` is just a shorthand for the NVMe-oF format mentioned above. We will be using both storage classes to run a single replica followed by a two way replica AKA mirror.  We use the following YAML to create the volume.\n\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: ms-volume-claim\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 100G\n storageClassName: nvmf\n\nAfter creating the PVC, Mayastor’s control plane creates a CRD, “Mayastor Volume” (MSV), that contains additional information about the corresponding volume.  Using kubectl describe msv -n mayastor we get:\n\n Name: ba081dc3-46db-445b-969c-7e5245aba146\n Namespace: mayastor\n Labels: \n Annotations: \n API Version: openebs.io/v1alpha1\n Kind: MayastorVolume\n Metadata:\n Creation Timestamp: 2020-09-11T08:49:30Z\n Generation: 1\n Managed Fields:\n API Version: openebs.io/v1alpha1\n Fields Type: FieldsV1\n fieldsV1:\n f:spec:\n .:\n f:limitBytes:\n f:preferredNodes:\n f:replicaCount:\n f:requiredBytes:\n f:requiredNodes:\n f:status:\n .:\n f:nexus:\n .:\n f:children:\n f:deviceUri:\n f:state:\n f:node:\n f:reason:\n f:replicas:\n f:size:\n f:state:\n Manager: unknown\n Operation: Update\n Time: 2020-09-11T08:51:18Z\n Resource Version: 56571\n Self Link: /apis/openebs.io/v1alpha1/namespaces/mayastor/mayastorvolumes/ba081dc3-46db-445b-969c-7e5245aba146\n UID: 94e11d58-fed9-44c9-9368-95b6f0712ddf\n Spec:\n Limit Bytes: 0\n Preferred Nodes:\n Replica Count: 1\n Required Bytes: 100000000000\n Required Nodes:\n Status:\n Nexus:\n Children:\n State: CHILD_ONLINE\n Uri: bdev:///ba081dc3-46db-445b-969c-7e5245aba146\n Device Uri: nvmf://x.y.z.y:8420/nqn.2019-05.io.openebs:nexus-ba081dc3-46db-445b-969c-7e5245aba146\n State: NEXUS_ONLINE\n Node: atsnode3\n Reason:\n Replicas:\n Node: node3\n Offline: false\n Pool: pool-atsnode3\n Uri: bdev:///ba081dc3-46db-445b-969c-7e5245aba146\n Size: 100000000000\n State: healthy\n Events: \n\n## Results single replica\n\n![Table (% difference of Mayastor overhead)](/images/blog/mayastor-nvme2.png)\n\n![Chart](https://lh5.googleusercontent.com/whpgDl_Id_oo4tUdl7RZDv-C1Uq2ZfvM6Eh7KXbwNkNTu5Mki14meunBgF1PMWMnWLoccSGgHqCfRKXgQpJTfQG42NaS0GkwWRCuNpWGh7znOhqQ94aJXiCODJkzNUs9-G2ucqMJ)\n\nFrom the results we can see that we are very close to the performance of the local device. To be sure we can put it in the right perspective, the difference between the experiments here is that with the baseline, the workload was local. With repl=1 we use the same NVMe device but export it through our pool layer (and thus provide volume management), but also go over the network.\n\n## Results 2 replicas (mirror)\n\nWe are going to repeat the same test, this time, we will use two replicas. As we now have double the disks bandwidth, we expect to see that the read performance will go up. For writes, however, we actually expect a drop in performance, because we must do each write to both disks before we can acknowledge the IO.  Note that Mayastor does not cache - if you read the blog referenced above from Richard Elling you can learn why caching seems to be falling out of favor for use cases in which millions of IOPS are desired.\n\n![Mayastor nvme](/images/blog/mayastor-nvme3.png)\n\n[Chart](https://lh4.googleusercontent.com/GJ7c_cZ6vuDxd9-jAnU3XxAc8L0idA9sscz2JB5XVa0taj8v56H6MSIFB56XqPQzQsy_p49-yHlwhCB8SVjZ05YfT0oRdlFt0EMBze1IDrCioqWgtWypidK9fBpb9p3ULI19Dhfa)\n\n## Wrapup\n\nWith the above experiments, we have demonstrated that with OpenEBS Mayastor we have built a foundational layer that allows us to abstract storage in a way that Kubernetes abstracts compute. While doing so, the user can focus on what's important -- deploying and operating stateful workloads. \n\nIf you’re interested in trying out Mayastor for yourself, instructions for how to setup your own cluster, and run a benchmark like **fio** may be found at [mayastor.gitbook.io/](https://mayastor.gitbook.io/introduction/).\n\nAnd if you are a Kubera Propel user, you’ll find that we’ve productized some of the benchmarking above so that platform teams and other users can programmatically use benchmarks in their decisions about workload placement. We are working with a number of users about operating OpenEBS Mayastor / Kubera Propel at scale. Please get in touch if you have suggestions, feedback, ideas for interesting use cases and so on. Contributions of all kinds are welcome!\n","notHasFeatureImage":false,"slug":"mayastor-nvme-of-tcp-performance"},{"id":16,"title":"Mayastor Engine Reaches Beta","author":"Glenn Bullingham","author_info":"Director of Solution Architecture","date":"19-11-2020","tags":["OpenEBS"],"excerpt":"Mayastor, the storage engine by OpenEBS has reached the beta stage. Read the blog to know more.","content":"\nAs I write this, it is early November, and the winds of change are blowing. The United States has a new president. Here in the United Kingdom, Keats' days of mists and mellow fruitfulness are departing, replaced by a low sun and the first morning frosts. And at MayaData, we see the Mayastor project carefully but tenaciously emerging from alpha/pre-release into its beta phase. In fact, Mayastor now undergirds MayaData’s commercial offering for performance sensitive containerized workloads, called [Kubera Propel](https://mayadata.io/product).\n\n> ***“Beta Software: Software considered to be feature complete and substantially free of known major defects”***\n\nSignificant contributions over the past 18 months have seen the project raised from concept to working software. A major requirement of our MVP specification is that it should carry minimal performance overhead; Mayastor is intended to satisfy demands for performance at all levels of scale. At the beginning of Autumn, working in conjunction with Intel’s labs in the UK, we were able to validate that assertion; deployed in conjunction with the latest generation of Optane NVMe devices, the Mayastor data plane was found to introduce less than 6% overhead; you can read more about that benchmarking [here](https://openebs.io/blog/mayastor-nvme-of-tcp-performance/). Having addressed that performance criteria and the other principle MVP requirements, the Mayastor team at MayaData has begun to focus its contributions to the project on QA as we approach Beta and GA releases.\n\nIn particular, we’ve greatly increased end-to-end test coverage on Kubernetes. How MayaData tests Mayastor is something that I’ll elaborate upon in a forthcoming post.  However, suffice it to say customary suspects feature (Jenkins, mocha, nix, cargo test), whilst we’re also collaborating with our colleagues who maintain the [Litmus Chaos](https://litmuschaos.io/) project. By the time that you’re likely reading this, Mayastor-specific chaos tests should be available to all on [ChaosHub](https://hub.litmuschaos.io/).\n\n## Ease of Use, Perf, and Availability\n\nMayastor MVP requirements center on ease of use, performance, and availability. In the Beta phase and subsequent GA release, these will be realized as a CAS platform with full NVMe data path semantics, declarative configuration via CSI compliant dynamic provisioning, and N+1 synchronous mirroring of data at the persistent layer. This closely approaches functional parity with the current OpenEBS storage engines (cStor, Jiva, and Local PV), with snapshot and cloning functionality to be added in Q1 2021. Mayastor will also very soon be the recipient of a streamlined deployment and configuration experience, which is exclusive to this engine.\n\n## Try it Yourself\n\nIf you’re a member of the Kubernetes community looking to implement a platform-native storage solution in the new year, now is an ideal time to start evaluating Mayastor. The other venerable and respected engines of OpenEBS won’t be retiring overnight, but as full feature parity emerges, MayaData’s commercial products and services will on Mayastor as their default storage engine; we do recognize that some users will continue to prefer various flavors of Dynamic Local PV from OpenEBS - as a recent [blog](https://www.percona.com/blog/2020/10/01/deploying-percona-kubernetes-operators-with-openebs-local-storage/) from the CTO of Percona attests as do countless [Adopter.md case studies](https://github.com/openebs/openebs/blob/master/ADOPTERS.md) including that of the SaaS company Optoro, also a CNCF case study. Mayastor’s roadmap includes provisions for the inward migration of existing OpenEBS users. It’s an equally opportune moment to [consider becoming a contributor](https://github.com/openebs/Mayastor/issues/new/choose) to the project yourself.\n\nTo help with Mayastor onboarding as we prepare to go to full steam, we’re putting together a new documentation site over at[ GitBook](https://mayastor.gitbook.io/introduction/), which includes a comprehensive quick-start deployment guide (developer docs will remain, at least for now, with the OpenEBS/Mayastor GitHub repository). We’re also holding [Office Hours at Kubecon NA 2020 this month](https://kccncna20.sched.com/?searchstring=OpenEBS&iframe=no&w=&sidebar=&bg=), and we’d love to see you there.\n\nIf you’d like to try Mayastor from the source - you can do so, of course, from the GitHub repositories. If you’d like to also try out management utilities, including a cool management interface and available 24x7 support - please take a look at [Kubera Propel](https://go.mayadata.io/register-for-kubera-chaos-and-propel-technical-preview). A free forever for individual use tier is available.\n\n## Conclusion\n\nIt is a propitious time for MayaData and Mayastor - and for data on Kubernetes more broadly. If you have always wanted to run workloads on Kubernetes but were put off by the stories of performance challenges, you can now move forward with confidence. Kubernetes enabled storage with the help of Mayastor performs faster than that of traditional shared everything storage while retaining the ease of use, open source community, and Kubernetes semantics for which OpenEBS has become famous.\n","notHasFeatureImage":false,"slug":"mayastor-engine-reaches-beta"},{"id":17,"title":"Migrate CSPIs to a different node by moving the disks","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"04-11-2020","tags":["OpenEBS"],"excerpt":"Step by step guide to migrate CStorPoolInstances from one node to different nodes by moving the set of underlying disks","content":"\nThis blog describes steps to migrate CStorPoolInstances from one node to different nodes by **moving the set of underlying disks to a new node that participates in the pool provisioning**. There were a couple of use cases where this feature can be helpful:\n\n1. Scaling down and scaling up nodes in the cluster(in a cloud environment) by retaining external volumes(for cost savings).\n2. Replacing failed storage nodes with new nodes by attaching the same old disks to the new node.\n\n**Steps to migrate the CSPI to different node:**\n\n1. Detach the disks belonging to the CSPI that you wish to migrate from the node and attach it to the new node. If you are using a cloud platform, check on their documentation, or ask the administrator about the steps to do this.\n2. Change the node selector in the CSPC YAML (next section describes how to do this).\n\n![](https://lh4.googleusercontent.com/XTwKu6lE3lyoZ3cHRO9HNJGUaTOoGfE-OWGuscrmukbxEKJNPSaEqxUPbbNnnc3dcD-Aybc2_AF0y2Scf0QBxSDG_f9QZWRu67sXZjoMKO6nymhgelEWfDzPjfGKi4D9UwLBaN0D)\n\n**Existing setup**:\n\nI have a three-node cluster with CSPC and CSI volumes already provisioned(To create CSPC pools and CSI volume click [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md#quickstart)). Here is detailed information:\n\n**Cluster details**:\n\n Kubernetes Cluster: AWS\n Kubernetes Version: v1.15.9\n OpenEBS Version: 2.2.0\n\n**Node and BlockDevice details**: Attached three disks to three nodes(each node has one disk)\n\n Kubectl get nodes\n\n NAME STATUS ROLES AGE VERSION\n ip-192-168-29-151 Ready 16m v1.15.9\n ip-192-168-36-89 Ready 8h v1.15.9\n ip-192-168-74-129 Ready 8h v1.15.9\n\n Kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-36-89 10737418240 Claimed Active\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-74-129 10737418240 Claimed Active\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-29-151 10737418240 Claimed Active\n\n**CSPC Manifest**: Applied following CSPC manifest to provision cStor pools\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorPoolCluster\n metadata:\n name: cstor-disk-cspc\n namespace: openebs\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-74-129\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c2c846cce1befec7fbdcbae254329b0b\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-36-89\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-7d311a98255a454a717427b5c2d38426\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-29-151\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c608881cd3edbeab674a1aee7e0a1fc3\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n\nAfter applying the above CSPC manifest, the following three CStorPoolInstances(CSPI) were created.\n\n kubectl get cspi -n openebs\n\n NAME HOSTNAME FREE CAPACITY READONLY STATUS AGE\n cstor-disk-cspc-dvc2 ip-192-168-74-129 24100M 24111M false ONLINE 8h\n cstor-disk-cspc-f56z ip-192-168-36-89 24100M 24113200k false ONLINE 8h\n cstor-disk-cspc-q9yt ip-192-168-29-151 24100M 24113200k false ONLINE 8h\n\nNow everything looks good. After some time, the cluster has been scaled down **0** nodes and scaled back to **3** nodes. So after scaling operations following are new nodes in the cluster.\n\n Kubectl get nodes\n\n NAME STATUS ROLES AGE VERSION\n ip-192-168-14-90 Ready 118s v1.15.9\n ip-192-168-49-43 Ready 5m55s v1.15.9\n ip-192-168-94-113 Ready 4m6s v1.15.9\n\nAttached old disks that participated in pool creation to new nodes, and the following is blockdevice output.\n\n Kubectl get bd -n openebs\n\n NAME NODENAME SIZE CLAIMSTATE STATUS\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-49-43 10737418240 Claimed Active\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-94-113 10737418240 Claimed Active\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-14-90 10737418240 Claimed Active\n\nFrom the above and previous output following are blockdevice mappings with zn old node and new node:\n\n Blockdevice Name Old Node New Node\n blockdevice-7d311a98255a454a717427b5c2d38426 ip-192-168-36-89 ip-192-168-49-43\n blockdevice-c2c846cce1befec7fbdcbae254329b0b ip-192-168-74-129 ip-192-168-94-113\n blockdevice-c608881cd3edbeab674a1aee7e0a1fc3 ip-192-168-29-151 ip-192-168-14-90\n\nOpenEBS **NodeDiskManager**(NDM) will automatically update the details in blockdevice CRs when the disks migrate to a new node. Based on the above output, update the CSPC manifest with new **nodeSelector** values.\n\n**Updated CSPC Manifest**:\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorPoolCluster\n metadata:\n name: cstor-disk-cspc\n namespace: openebs\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-94-113\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c2c846cce1befec7fbdcbae254329b0b\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-49-43\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-7d311a98255a454a717427b5c2d38426\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n - nodeSelector:\n kubernetes.io/hostname: \"ip-192-168-14-90\"\n dataRaidGroups:\n - blockDevices:\n - blockDeviceName: \"blockdevice-c608881cd3edbeab674a1aee7e0a1fc3\"\n poolConfig:\n dataRaidGroupType: \"stripe\"\n\nOnce the CSPC manifest is updated then CSPIs will automatically migrate to the new node (which can be verified using **kubectl get cspi -n openebs**).\n\n kubectl get cspi -n openebs\n\n NAME HOSTNAME FREE CAPACITY READONLY STATUS AGE\n cstor-disk-cspc-dvc2 ip-192-168-94-113 24100M 24111M false ONLINE 8h\n cstor-disk-cspc-f56z ip-192-168-49-43 24100M 24113200k false ONLINE 8h\n cstor-disk-cspc-q9yt ip-192-168-14-90 24100M 24113200k false ONLINE 8h\n\n**Note:** Along with CStorPoolInstance migration, CStorVolumeReplicas belongs to CSPI will also migrate automatically.\n","notHasFeatureImage":false,"slug":"migrate-cspis-to-a-different-node-by-moving-the-disks"},{"id":18,"title":"OpenEBS Backup/Restore for ZFS-LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"27-10-2020","tags":["OpenEBS"],"excerpt":"Overview of how to use Velero Backup/Restore plugin for ZFS-LocalPV to protect it against data loss.","content":"\n## Overview: OpenEBS Backup/Restore for ZFS-LocalPV\n\n**Backup** is the process of copying the data to a different/remote location to protect against accidental or corruption or any other type of data loss. Restore is the process of getting back the data from the backup. In this blog, I will discuss how we can use *Velero Backup/Restore* plugin for ***ZFS-LocalPV*** to protect it against data loss.\n\n### Pre-requisites\n\nWe should have installed the ZFS-LocalPV 1.0.0 or later version for Backup and Restore, see my previous[ blog](/blog/openebs-dynamic-volume-provisioning-on-zfs) for the steps to install the ZFS-LocalPV driver.\n\n### Setup\n\n**1.Install Velero CLI**\n\nDownload the 1.5 or later binary for ZFS-LocalPV. For Linux on amd64, we need to download below\n\n wget\n https://github.com/vmware-tanzu/velero/releases/download/v1.5.1/velero-v1.5.1-linux-amd64.tar.gz\n\nExtract the tarball:\n\n tar -xvf velero-v1.5.1-linux-amd64.tar.gz\n\nMove the extracted velero binary to somewhere in your $PATH (/usr/local/bin for most users).\n\nSee the detailed steps[ here](https://velero.io/docs/v1.5/basic-install/).\n\n**2.Deploy Velero**\n\nWe will be using minio for storage purpose in this blog, we need to setup the credential file first\n\n $ cat /home/pawan/velero/credentials-minio\n [default]\n aws_access_key_id = minio\n aws_secret_access_key = minio123\n\nWe can install Velero by using below command\n\n $ velero install --provider aws --bucket velero --secret-file /home/pawan/velero/credentials-minio --plugins velero/velero-plugin-for-aws:v1.0.0-beta.1 --backup-location-config region=minio,s3ForcePathStyle=\"true\",s3Url=http://minio.velero.svc:9000 --use-volume-snapshots=true --use-restic\n\nWe have to install the velero 1.5 or later version of velero for ZFS-LocalPV.\n\n**3.Deploy MinIO**\n\nDeploy the MinIO for storing the backup:-\n\n $ kubectl apply -f\n https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/sample/minio.yaml\n\nThe above MinIO uses tmp directory inside the pod to store the data for the demonstration purpose, so when restart happens, the backed up data will be gone. We should change the above YAML to use persistence storage to store the data when deploying it for the production.\n\nCheck the Velero Pods are UP and Running\n\n $ kubectl get po -n velero\n NAME READY STATUS RESTARTS AGE\n minio-d787f4bf7-xqmq5 1/1 Running 0 8s\n minio-setup-prln8 0/1 Completed 0 8s\n restic-4kx8l 1/1 Running 0 69s\n restic-g5zq9 1/1 Running 0 69s\n restic-k7k4s 1/1 Running 0 69s\n velero-7d9c448bc5-j424s 1/1 Running 3 69s\n\n**4.Setup OpenEBS Plugin**\n\nWe can Install the Velero Plugin for ZFS-LocalPV using the below command\n\n velero plugin add openebs/velero-plugin:2.2.0\n\nWe have to install the velero-plugin 2.2.0 or later version, which has the support for ZFS-LocalPV. Once the setup is done, we can go ahead and create the backup/restore.\n\n**5.Create the VSL**\n\nThe VSL(Volume Snapshot Location) has information about where the snapshot should be stored. To create the Backup/Restore, we can create the Volume Snapshot Location by applying the below YAML:\n\n apiVersion: velero.io/v1\n kind: VolumeSnapshotLocation\n metadata:\n name: default\n namespace: velero\n spec:\n provider: openebs.io/zfspv-blockstore\n config:\n bucket: velero\n prefix: zfs\n namespace: openebs # this is the namespace where ZFS-LocalPV creates all the CRs, passed as OPENEBS_NAMESPACE env in the ZFS-LocalPV deployment\n provider: aws\n region: minio\n s3ForcePathStyle: \"true\"\n s3Url: http://minio.velero.svc:9000\n\nHere, we have to provide the namespace, which we have used as OPENEBS_NAMESPACE env while deploying the ZFS-LocalPV. The ZFS-LocalPV Operator yamls uses “openebs” as the default value for OPENEBS_NAMESPACE env. Verify the volumesnapshot location:\n\n kubectl get volumesnapshotlocations.velero.io -n velero\n\n### Create Backup\n\nWe can use the below Velero command to create the backup:\n\n velero backup create my-backup --snapshot-volumes --include-namespaces= --volume-snapshot-locations=default --storage-location=default\n\nwe can add all the namespaces we want to be backed up in a comma-separated format in --include-namespaces parameter. We have to provide the VSL that we have created in --volume-snapshot-locations parameter.\n\nWe can check the backup status using the velero backup get command:\n\n $ velero backup get\n NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR\n my-backup InProgress 2020-09-14 21:09:06 +0530 IST 29d default \n\nThe status InProgress means that the backup is in progress. Wait for it to be Completed.\n\nWe can also create a scheduled backup which will take the backup periodically. For example, to take the full backup at every 5 min, we can create the below schedule :\n\n velero create schedule schd --schedule=\"*/5 * * * *\" --snapshot-volumes --include-namespaces=, --volume-snapshot-locations=default --storage-location=default\n\n### Restore\n\nIf the application and its PVC has been deployed in a namespace, then we can use the below Velero command to create the backup of the entire namespace:\n\n velero restore create --from-backup my-backup --restore-volumes=true --namespace-mappings :\n\nThe above command will create the backup of everything that is there in the namespace provided as --include-namespaces argument. We can provide the namespace mapping if we want to restore in a different namespace as --namespace-mappings parameter. If namespace mappings are not provided, it will restore in the source namespace only where the original pod and pvc was present. Now we can check the restore status:\n\n $ velero restore get\n NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR\n my-backup-20200914211331 my-backup InProgress 0 0 2020-09-14 21:13:31 +0530 IST \n\nOnce the Status is Completed, we can check the pods in the destination namespace and verify that everything is up and running. We can also verify that the data has been restored.\n\n### Summary\n\nAs demonstrated in this blog, OpenEBS makes it easy to take the backup of the Kubernetes applications, which we can use to Restore as part of disaster recovery. In my next blog, I will talk about how we can take the incremental backup of the volumes, which is space optimized backup for ZFS-LocalPV\n\n## Important links\n\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n[https://velero.io/docs/](https://velero.io/docs/v1.5/basic-install/)\n","notHasFeatureImage":false,"slug":"openebs-backup/restore-for-zfs-localpv"},{"id":19,"title":"OpenEBS 2.2.0 - Enhancements And New Storage Capabilities","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"20-10-2020","tags":["OpenEBS"],"excerpt":"OpenEBS 2.2.0 is here! Read this post to learn about the new updates.","content":"\n### **OpenEBS 2.2.0 is here**\n\nWe are excited to announce yet another ***OpenEBS*** release that comes with new storage capabilities, control plane enhancements, bug fixes, and new APIs for the world’s fastest storage engine built on RUST, also known as Mayastor.\n\nOpenEBS has seen a wider adoption among the users, thanks to the vibrant and growing community. Like in most of the OpenEBS releases, this release responds to the feedback received in the community. If you want to learn more about the project roadmap, please browse the following link:\n[https://github.com/openebs/openebs/blob/master/ROADMAP.md](https://github.com/openebs/openebs/blob/master/ROADMAP.md)\n\nIncremental Backup and Restore in ZFS local PV and pool and volume migration in cStor are the major release milestones made into the release. The pool migration in cStor solves the use-case of replacing a bad node with a new node or sending a node for maintenance on on-premise clusters. The migration feature provides great value in cloud-managed Kubernetes clusters, too, e.g., GKE, where node reboots or voluntary scale down of nodes can cause the disks to get removed. \n\nThis release is also special due to the [*Hacktoberfest*](https://hacktoberfest.digitalocean.com/) festival and would like to give a shout out to first-time contributors [@didier-durand](https://github.com/didier-durand), [@zlymeda](https://github.com/zlymeda), [@avats-dev](https://github.com/avats-dev), and many more.\n\n### **Key Highlights of OpenEBS 2.2.0 Release:**\n\n- Mayastor aims to be the world’s fastest container attached storage and is currently in alpha. The release introduced block device enumeration feature via the gRPC API and enhancement around storage pool finalizers.\n- ZFS local PV has become a popular storage engine built on local storage design and provides powerful storage features like snapshots and clones, raw block volume, etc. It also supports day two operations like volume resize and backup and restore via the pluggable Velero interface.\nSupport for Incremental Backup and Restore by enhancing the OpenEBS Velero Plugin has been a significant highlight for ZFS local PV release. \nTo learn more about this, please refer to the document [here](https://github.com/openebs/zfs-localpv/blob/master/docs/backup-restore.md).\n- OpenEBS Velero plugin connects the Velero interface to the OpenEBS storage engines to deliver backup/restore functionality. Velero Plugin has been enhanced to restore ZFS Local PV into a different cluster or a different node in the cluster and use custom certificates for S3 object storage.\n- CStor went into beta in 2.0 and has been enhanced to migrate the storage pool from one node to another node. This will help with scenarios where a Kubernetes node can be replaced with a new node but can be attached with the block devices from the old node that contain cStor Pool and the volume data.\n- OpenEBS node disk manager helps in block device discovery and management in a Kubernetes cluster and powers storage engines like cStor. Support to exclude multiple devices that could be mounted as host filesystem directories has been added.\nAn issue where NDM could cause data loss by creating a partition table on an uninitialized iSCSI volume has also been fixed.\n\n### **Useful Links and Summary:**\n\nIf you are interested in knowing more details regarding the changes that made to this release, please visit the release note [link](https://github.com/openebs/openebs/releases/tag/v2.2.0). To try out OpenEBS, you can visit [https://openebs.io/docs](/docs) and follow the user guides.\n\nYou can visit the following link to learn more or experiment with Mayastor\n[https://github.com/openebs/mayastor](https://github.com/openebs/mayastor)\n\nYou can visit the following link to learn more or experiment with ZFS local PV\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n\nTo learn more about the new cStor CSPC API, please visit the following link:\n[https://github.com/openebs/cstor-operators](https://github.com/openebs/cstor-operators)\n\nIf you have any feedback, questions, or suggestions — please reach out to the community on the #openebs channel in the Kubernetes workspace or consider opening a relevant issue at [Github](https://github.com/openebs/openebs).\n","notHasFeatureImage":false,"slug":"openebs-2.2.0---enhancements-and-new-storage-capabilities"},{"id":20,"title":"Scaling up cStor Volume Replica","author":"Abhishek","author_info":"Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.","date":"07-10-2020","tags":["OpenEBS"],"excerpt":"OpenEBS provides volume replication through different storage engines. Learn how to scale up cStor Volume Replica.","content":"\nEven if a cluster is reliable, nodes can and do fail. Rebooting a node does not simulate a crash. There can be many reasons, such as catastrophic hardware failure, Operating System failure, or communication failure among the nodes. To overcome this hazardous situation, the Replication of volume becomes necessary.\n\nReplication is the process by which one or more volumes can be copied to maintain the significance of a cluster and to avoid data loss. OpenEBS provides volume replication through different storage engines. One of them is cStor Volume Replication.\n![Synchronous replication of data](https://lh5.googleusercontent.com/ijS24Ywabw-QkWWYbSLoOshGTi2SHZhdEFATaHIYbkNGK8lUq5SJrct6fNHfPjWcPTHGyvByS7uD1vYct2m5D6-HdRC2ZoMpS_c4Crw-9sREhPU-tXE8KAt-nWj7vYw99Ee_s1pE)\n#### Prerequisite for scaling up the replicas of cStor volume:\n\n- A cStor pool should be available, and replicas of this cStor volume should not be present on this cStor pool.\n- The OpenEBS version should be 1.3.0 or more.\n\n### Please follow the below steps for cStor Volume Replication:\n\nGet the StorageClass name using the following command:\n\n kubectl get sc\n\nExample Output:\n![](https://lh5.googleusercontent.com/lTma7ZqsAavmXEzGG_b4BXDMUEYXjFXf0xxnWgE70znfR_EzP3IorVFp0evkKoLMsBQ0D7gwOQxivB_bZxEcv2vhYZOe17k7mNyDBaPewTgiUdusrd3ow12ClBeQvZVmVzjDrdsI)\nThe storage class for cStor is ***openebs-sc-cstor***. Perform the following command to get the details of the corresponding StorageClass, which is used for creating the cStor volume :\n\n Kubectl get sc openebs-sc-cstor\n\nWe will get the Yaml file of the corresponding StorageClass ***openebs-sc-cstor***.\n![](https://lh5.googleusercontent.com/81DQJ-DhT3AKseMRfCZ4NpkmOPl2Tckm76jrUxE2eECY7lrejvNz3OjomFWmNiCRwm0L2seAWzmJJhe-8xcqFirBsEUedf2xzPN4NHq2RM2YYEZZv-iKpsE03j06EQi_D5kqnDCi)\nIn the Yaml above, We can see the Replica count Value is 1.\n\nGet the volume name using the following command:\n\n Kubectl get pvc\n\nGet the VOLUME name and use it in the following command to get the details of corresponding cStor volume. All commands are performed by considering the above PVC.\n\n kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=\n\nExample output:\n![](https://lh4.googleusercontent.com/FIOJchscq3lm7UJLwnk7i1oNne_RxhjIJzI3FMANxxkRhz4yWZAue-Wu1jD03ii2aMjtdDu3zr9C-0ZGaeazkvxb_JkGnxBBDza605w_p-v9BY1ER40f6DityHwimJvhvuAR8FcT)\nGet the details of existing cStor Volume Replica details using the following command:\n\n kubectl get cstorvolume -n openebs -l openebs.io/persistent-volume=pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8\n\nExample output:\n![](https://lh3.googleusercontent.com/68NvgkfD7audTNZN1QLt6SVw4OvN_B3MIlnFnWm8MfgDziiexFX2qeI3tX6H1TCJJgrCA8b-nZQJoM6hx1QoYWOv4q74tKwB7nrZLc9xdluXRCvWTpj-sU6sIv7aJ0AMgL3rr1AR)\nPerform the following command to get complete details of the existing cStor volume replica:\n\n kubectl get cvr -n openebs -l openebs.io/persistent-volume=pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8\n\nGet the available cStor Pools for creating new cStor volume replica. The following command will get the other associated cStor pools details:\n\n kubectl get csp -l openebs.io/storage-pool-claim=cstor-disk-pool | grep -v cstor-disk-pool-hgt4\n\nExample Output:\n![](https://lh6.googleusercontent.com/lcbO830nSZgValr-I4ci7FHRa6Qvqf3eG-bycWHHAniRD8mb8dwRHOwxeVObFqj4FqvXbNkb_oZUdWhMgAQuHvU1pYDecvWXhDetYGdJADBQhWfzMuwJm4d9Ywgg6bAKkj-Sd79a)\nFrom the above example output, there are 2 cStor pools available, i.e., ***cstor-disk-pool-2phf*** and ***cstor-disk-pool-zm8l***. So it is possible to scale up the current volume replica count to 3 from 1. If there are no cStor pools available to perform volume replica scale-up, then follow the [steps](/docs/deprecated/spc-based-cstor#expanding-cStor-pool-to-a-new-node) to create a new cStor pool by updating existing SPC configuration.\n\nPerform the following command to get the details of the cStor Pool where new replica will be created:\n\n kubectl get csp -n openebs cstor-disk-pool-2phf -oyaml\n\nNote down following parameters from the output:\n\n- metadata.labels.cstorpool.openebs.io/name\n- metadata.labels.cstorpool.cstorpool.openebs.io/uid\n- metadata.annotations.cstorpool.openebs.io/hostname\n\nThe sample CVR Yaml is provided below:\n![](https://lh3.googleusercontent.com/JePqVqyIryf396SEkCf9NoS3kmPDXM0huqehkN3kX5f-eE7nX3-mCr42xriJeDKSNRgfVxSeQG_SUHkbqEZS4ktIzzcJ8VKCsFXuz4VhtdXpikLADE3eJdkgwH3zFd5PXRPfYc70)\nApply the updated CVR YAML spec to create the new replica of cStor volume using the following command:\n\n kubectl apply -f cvr.yaml\n\nExample Output:\n![](https://lh5.googleusercontent.com/JElB0d8zFXHoUh6wM0QpAshOmYbVXOvH5RIR9UjJ_svM67ZR2pq6cQ4ckrq0Qw6ACpRnOqO-6nUbvLUrDhFKvgZxjrh-ke0VHnKW-pR2oyzkgXdQuRATSwy9EVN19G458ZyR_9Xd)\nVerify if new CVR is created successfully using the following command:\n\n kubectl get cvr -n openebs\n\nExample output:\n![](https://lh4.googleusercontent.com/ql9j6Zcod6DT1vKhJrlJJaxk4YUN8Mf_o7LT3e-fBjjoybINByEwwDS5fln6K5BEJGW6vFfE8h2JA_2tFvQY5PQKo62eJvQfTE5j5JwECIz2oO3u_ypKHWRylL3gmU4KYlo4axtU)\nFrom the above output, a new replica of the cStor volume is created, and STATUS is showing as Offline.\n\nUpdate Desired Replication Factor in cStor volume with a new replica count. This can be updated by editing corresponding cStor volume CR YAML.\n\n kubectl edit cstorvolume pvc-3f86fcdf-02f6-11ea-b0f6-42010a8000f8 -n openebs\n\nThe following is the snippet of updated cStor volume CR YAML:\n![](https://lh3.googleusercontent.com/lAisXwgequP2MyeCw1cVuwUYFG9G9L5U88olJ2CjbjIOpHjlMwn-K8p11ktaCjQfxK-u5EL-ebpZofD0W_LOKmfFa-wW3eTLtBpqSt7EPYvz5rQciYeaFdT6_7PCsJkdxPVHZCVg)\n\nIn the above snippet, the desiredReplicationFactor is updated to 2 from 1. Example output:\n![](https://lh6.googleusercontent.com/uBkJft958gfjATk070ZFZOMXaq7Sb1xnd5lBVMa2sKuXo-nxwrRxQS58TPgdpoLjMuMHvT4LwPscxPdT6kgwpaDVSraLmsNwWhfanMUrNVO72K8WgxwT3_or4EdzqQkWBgI-Ka84)\nVerify if the rebuilding has started on the new replica of the cStor volume. Once rebuilding has been completed, it will update its STATUS as Healthy. Get the latest status of the CVRs using the following command:\n\n kubectl get cvr -n openebs\n\nExample output:\n![](https://lh6.googleusercontent.com/1KjmeLgtvoFcBh0vVmB0iwj_gjo-Tkd3vVTTmaw3OaREY9KbvDUQLqyEu0Hj_aYKDpTIRSDVG2sOrTPMczJAPASlzFitSHDyocPV4Bb6IgajW-ArUpDKhi8StFesnHYZrUc3X9DJ)\n","notHasFeatureImage":false,"slug":"scaling-up-cstor-volume-replica"},{"id":21,"title":"Hacktoberfest 2020 - Contribute to OpenEBS","author":"MayaData Marketing","author_info":"Mayadata Marketing Team","date":"30-09-2020","tags":["OpenEBS"],"excerpt":"Hacktoberfest 2020 is almost here. Contribute to open source project, OpenEBS, to win exciting swag.","content":"\n### Hacktoberfest returns! Contribute to OpenEBS and win exciting swag\n\nThe seventh annual [***Hacktoberfest***](https://hacktoberfest.digitalocean.com/) celebration is almost here, and we at *OpenEBS* are happy to be participating in the contest once again. In August 2017, the OpenEBS community began growing and building a strong foundation for an open source project.\n\nWe were first introduced to *Hacktoberfest* by friends and peers at the DigitalOcean Bangalore Meetup and were immediately interested in participating. We enlisted OpenEBS as one of the projects participating in Hacktoberfest 2017. We were pleasantly surprised by the participation and enthusiasm that Hacktoberfest attracts from developers around the world. The PRs have been at their peak during Hacktoberfest.\n\n![](https://lh4.googleusercontent.com/Og_t8KLCiRni_LS66bpJsonSXMjcoAX671c8a2LD7ZjbkVdYZgZCRFq47sDC7hsEZt6qcaoCJPZi_gm2FnKmuzMvlg4UZAQKofU0agH2Z11TRmw6vBCQ8u3ssGfre75BN9OV-vOO)\n\n### Get started with OpenEBS this Hacktoberfest\n\nFollowing the smashing success we had when we participated in the event last few years, we’re going to do the same this year! MayaData makes it more exciting to participate in **Hacktoberfest** by running multiple Meetups throughout the month and helping contributors to get started with their favorite areas (in any of the programming languages) like website development and documentation enhancements. \n\nTo top it all, there are exciting prizes to be won and every contribution deserves an additional swag from MayaData. Read [this blog](https://blog.mayadata.io/openebs/experience-with-openebs-in-this-hacktoberfest) by Aswath K, one of last year’s weekly winners, who writes about his experience with OpenEBS in Hacktoberfest 2019.\n\n![](https://lh6.googleusercontent.com/2POqPppb7pyGM0OWwl_LlkHzwz-DSWXMMggxIeNCXvsU6EVVmNHdiIzIoTw23-ceK9R5iBleFMGiK-lw9JLtCP5VVjFGQS1QhIOXbpQhtvku5Gp5aCw4Eul_r6JcM-o0WuVZRZmj)\n\n### How do I contribute to OpenEBS?\n\nThat is an excellent question! OpenEBS is a Kubernetes native Container Attached Storage (CAS) that simplifies running Stateful workloads on Kubernetes. It is built on Microservices architectural patterns, fully automated development, and build pipelines.\n\nOpenEBS has several components that need your help from simple fixes like adding GitHub issue templates, to enhancing the components. These are developed using Go, Rust, Python, Javascript, Ansible, and many more interesting tools. OpenEBS is also a great way to start your journey into the exciting world of storage, containers, and Kubernetes.\n\nThe [architecture overview document ](https://github.com/openebs/openebs/blob/master/contribute/design/README.md) is a great place to start learning and picking up a component that speaks to your passion. You could start your first contribution by enhancing that document itself for providing more clarity.\n\nThere are many other [good first issues](https://github.com/search?q=org%3Aopenebs+is%3Aissue+label%3A%22good+first+issue%22) to pick from.\n\nContributions can be anything from creating issues, improving user and contributor documents, enhancing build and docker tools, fixing and enhancing code, or unit tests and e2e tests. If you are unsure where to start, begin a discussion with the contributors on [GitHub Discussions](https://github.com/openebs/openebs/discussions) or by joining [OpenEBS Community on Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F).\n\n### Will there be swag?\n\nYes. A big fat YES!\n\nThe official [Hacktoberfest](https://hacktoberfest.digitalocean.com/) will be giving away free t-shirts to every person making four pull requests to open source repositories during October, as well as limited-edition Hacktoberfest stickers to everyone who participates.\n\nOn top of this, you will also be able to get some exclusive and limited OpenEBS swag. When your PR to any OpenEBS repository is merged, we will contact you to fill out a form to send a special edition swag designed for Hacktoberfest.\n\nNot only this but by becoming a top weekly contributor, you’ll be able to grab even more swag.\n\nPrizes will be sent to quality contributions. The best PR will win a grand prize. Stay tuned to find out more.\n\nSo, what are you waiting for! Go get your git on and start contributing - we can’t wait to receive your PR.\n\nHappy hacking!\n\n### Getting Started:\n\n1. [https://hacktoberfest.digitalocean.com/](https://hacktoberfest.digitalocean.com/)\n2. Join [OpenEBS Community on Kubernetes Slack](https://kubernetes.slack.com/?redir=%2Fmessages%2Fopenebs%2F)\n3. Checkout the [OpenEBS Contributing guide](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md)\n4. Learn about the [architecture and components](https://github.com/openebs/openebs/blob/master/contribute/design/README.md) of OpenEBS\n5. Create new issues for your contribution or pick one of the existing issues from [https://github.com/openebs/openebs/issues](https://github.com/openebs/openebs/issues)\n","notHasFeatureImage":false,"slug":"hacktoberfest-2020---contribute-to-openebs"},{"id":22,"title":"OpenEBS StorageClasses For ZFS-LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"09-09-2020","tags":["OpenEBS","LocalPV","ZFS"],"excerpt":"In this blog, I will discuss various storage classes we can use to dynamically provision the volumes backed by ZFS-LocalPV Storage Pool.","content":"\nIn this blog, I will discuss various storage classes we can use to dynamically provision the volumes backed by ZFS Storage Pool.\n\nPlease read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the *ZFS-LocalPV*. Alternatively, you can also go through the [README](https://github.com/openebs/zfs-localpv/blob/master/README.md) section of the [*ZFS-LocalPV* repository](https://github.com/openebs/zfs-localpv).\n\n### **StorageClass Backed by ZFS Dataset**\n\nWe can create a StorageClass with the fstype as “zfs”. Here, the ZFS-LocalPV driver will create a ZFS dataset for the persistence storage. The application will get a dataset for the storage operation. We can also provide recordsize, compression, or dedup property in the StorageClass. The dataset will be created with all the properties mentioned in the StorageClass:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n recordsize: \"4k\"\n thinprovision: \"no\"\n fstype: \"zfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nWe have the thinprovision option as “no” in the StorageClass, which means that do reserve the space for all the volumes provisioned using this StorageClass. We can set it to “yes” if we don’t want to reserve the space for the provisioned volumes.\n\nThe allowVolumeExpansion is needed if we want to resize the volumes provisioned by the StorageClass. ZFS-LocalPV supports online volume resize, which means we don’t need to scale down the application. The new size will be visible to the application automatically.\n\nOnce the storageClass is created, we can go ahead and create the PVC and deploy a pod using that PVC.\n\n### **StorageClass Backed by ZFS Volume**\n\nThere are a few applications that need to have different filesystems to work optimally. For example, Concourse performs best using the “btrfs” filesystem ([https://github.com/openebs/zfs-localpv/issues/169](https://github.com/openebs/zfs-localpv/issues/169)). Here we can create a StorageClass with the desired fstype we want. The ZFS-LocalPV driver will create a ZVOL, which is a raw block device carved out from the mentioned ZPOOL and format it to the desired filesystem for the applications to use as persistence storage backed by ZFS Storage Pool:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: opeenbs-zfspv\n parameters:\n volblocksize: \"4k\"\n thinprovision: \"yes\"\n fstype: \"btrfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nHere, we can mention any fstype we want. As of 0.9 release, the driver supports ext2/3/4, xfs, and btrfs fstypes for which it will create a ZFS Volume. Please note here, if fstype is not provided in the StorageClass, the k8s takes “ext4\" as the default fstype. Here also we can provide volblocksize, compression, and dedup properties to create the volume, and the driver will create the volume with all the properties provided in the StorageClass.\n\nWe have the thinprovision option as “yes” in the StorageClass, which means that it does not reserve the space for all the volumes provisioned using this StorageClass. We can set it to “no” if we want to reserve the space for the provisioned volumes.\n\n### **StorageClass for Sharing the Persistence Volumes**\n\nBy default, the ZFS-LocalPV driver does not allow Volumes to be mounted by more than one pod. Even if we try to do that, only one Pod will come into the running state, and the other Pod will be in ContainerCreating state, and it will be failing on the mount.\n\nIf we want to share a volume among multiple pods, we can create a StorageClass with the “shared” option as “yes”. For this, we can create a StorageClass backed by ZFS dataset as below :\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"zfs\"\n shared: \"yes\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nOr, we can create the StorageClass backed by ZFS Volume for sharing it among multiple pods as below :\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"ext4\"\n shared: \"yes\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nHere, we have to note that all the Pods using that volume will come to the same node as the data is available on that particular node only. Also, applications need to be aware that the volume is shared by multiple pods and should synchronize with the other Pods to access the data from the volume.\n\n### **StorageClass With k8s Scheduler**\n\nThe ZFS-LocalPV Driver has its own scheduling logic, where it creates the volume where the ZFS Pool is less loaded with the volumes. Here, it just checks the volume count and creates the volume where less volume is configured in a given ZFS Pool. It does not account for other factors like available CPU or memory while making scheduling decisions. So if you want to use node selector/affinity rules on the application pod or have CPU/Memory constraints, the Kubernetes scheduler should be used. To make use of Kubernetes scheduler, we can set the volumeBindingMode as WaitForFirstConsumer in the storage class:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n fstype: \"zfs\"\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n volumeBindingMode: WaitForFirstConsumer\n\nHere, in this case, the Kubernetes scheduler will select a node for the POD and then ask the ZFS-LocalPV driver to create the volume on the selected node. The driver will create the volume where the POD has been scheduled.\n\n### **StorageClass With Custom Node Labels**\n\nThere can be a use case where we have certain kinds of ZFS Pool present on certain nodes only, and we want a particular type of application to use that ZFS Pool. We can create a storage class with `allowedTopologies` and mention all the nodes there where that pool is present:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: nvme-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: openebs.io/nodename\n values:\n - node-1\n - node-2\n\nHere we can have ZFS Pool of name “zfspv-pool” created on the nvme disks and want to use this high performing ZFS Pool for the applications that need higher IOPS. We can use the above StorageClass to create the PVC and deploy the application using that.\n\nThe ZFS-LocalPV driver will create the Volume in the Pool “zfspv-pool” present on the node with fewer of volumes provisioned among the given node list. In the above StorageClass, if there provisioned volumes on node-1 are less, it will create the volume on node-1 only. Alternatively, we can use `volumeBindingMode: WaitForFirstConsumer` to let the k8s select the node where the volume should be provisioned.\n\nThe problem with the above StorageClass is that it works fine if the number of nodes is less, but if the number of nodes is huge, it is cumbersome to list all the nodes like this. In that case, what we can do is, we can label all the similar nodes using the same key value and use that label to create the StorageClass.\n\n pawan@pawan-master:~/pawan$ kubectl label node pawan-node-2 openebs.io/zpool=nvme\n node/pawan-node-2 labeled\n pawan@pawan-master:~/pawan$ kubectl label node pawan-node-1 openebs.io/zpool=nvme\n node/pawan-node-1 labeled\n\nNow, restart the ZFS-LocalPV Driver (if already deployed otherwise, please ignore) so that it can pick the new node label as the supported topology.\n\n $ kubectl delete po -n kube-system -l role=openebs-zfs\n\nNow, we can create the StorageClass like this:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: nvme-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n allowedTopologies:\n - matchLabelExpressions:\n - key: openebs.io/zpool\n values:\n - nvme\n\nHere, the volumes will be provisioned on the nodes which has label “openebs.io/zpool” set as “nvme”.\n\n### **Conclusion :**\n\nWe can set up different kinds of StorageClasses as per our need, and then we can proceed with PVC and POD creation. The driver will take the care of honoring the requests put in the PVC and the StorageClass.\n\nI hope you found this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"openebs-storageclasses-for-zfs-localpv"},{"id":23,"title":"Handling node down/cluster scaling on ZFS LocalPV backed workloads","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"01-09-2020","tags":["Kubernetes"],"excerpt":"Step-by-step blog on how MySQL app deployment runs on OpenEBS ZFS LocalPV device, handled when a node down/cluster scale down situation happens in GKE cluster","content":"\nKubernetes is increasingly used for running production-grade stateful services. Organizations are making progress on a containerized form of their production workloads for running in Kubernetes. There are already solutions available for the containerized version of stateful applications, network, storage, etc.\n\nAs everyone knows, OpenEBS is one of the leading containerized storage solutions for Kubernetes, and it is a growing Sandbox project in CNCF. MayaData is the primary maintainer and contributor of OpenEBS along with other companies. MayaData also contributed another open source project, Litmus, into CNCF, which does mostly Chaos engineering in Kubernetes, which helps SREs and developers to do all kinds of testing of their applications and components in Kubernetes before going into Production.\n\nA persistent storage solution for running any stateful application is a must requirement, be it a **Deployment** or **StatefulSet**. OpenEBS provides various storage engines, and each storage engine is suitable for specific applications or workloads. Some engines provide storage level synchronous replication, capable of taking snapshots and cloning, backup and restore, volume expansion, CSI complaint, performance-oriented, etc. So choosing the engine based on the workload requirement is an important activity.\n\nOpenEBS provides dynamic provisioning of ***ZFS LocalPV*** using external device/devices. OpenEBS ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. Using a ***ZFS Local PV*** has the following advantages   as opposed to Kubernetes native Local PV backed by direct-attached devices:\n\n- Sharing of the devices among multiple application pods.\n- Enforcing quota on the volumes makes sure the pods don’t consume more than their capacity.\n- Ability to take snapshots of the LocalPV\n- Ability to sustain the disk failures — using the ZPOOL RAID functionality\n- Ability to use data services like compression and encryption.\n- Ability to resize the PV capacity.\n\nIn this article, we provisioned a MySQL deployment on an ***OpenEBS ZFS LocalPV*** device dynamically.\n\nThis article is a step-by-step instruction. We will mention how a MySQL application deployment running on the OpenEBS ZFS LocalPV device volume is getting handled when a Node down scenario or a cluster scale down situation happens in the GKE cluster. In GKE and some other managed clusters like EKS, the node name will change if the cluster undergoes a scale down and scale up operation has performed. So the application running on the OpenEBS ZFS LocalPV device will not be able to attach to the new node since the corresponding PV and ZFS Volume has a binding of volume node affinity. We need to update the new node name details in PV and ZFS volume, where the disk got attached.\n\nIn this article, we are discussing the steps that need to be performed to make the application into a running state when a Node down / scale down cluster scenario has happened. This situation is usually required in case of Managed clusters where the Node name will get changed during this scenario. As stated earlier, the following approach works fine for both Deployment type and StatefulSet type, but ensure that the next steps are correctly performed. Let’s start with the step by step instructions once you have scaled up the cluster after a scale down scenario.\n\n1. Verify all nodes are now in Ready state.\n ```\n $ kubectl get node\n NAME STATUS ROLES AGE VERSION\n gke-openebs-mysql-default-pool-dd23ce6b-f8rd Ready 24m v1.16.13-gke.1\n gke-openebs-mysql-default-pool-dd23ce6b-lwr3 Ready 24m v1.16.13-gke.1\n gke-openebs-mysql-default-pool-dd23ce6b-zzqx Ready 24m v1.16.13-gke.1\n ```\n2. Label all the nodes with the same custom label used in the nodeSelector field in the STS app. In my case, there is no custom node label used in application deployment. So we are skipping this step.\n\n3. Attach the disk randomly to any node in the same zone.\n \n ```\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-j894 --disk mysql-disk1 --device-name mysql-disk1 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-prv2 --disk mysql-disk2 --device-name mysql-disk2 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-dd23ce6b-tf5j --disk mysql-disk3 --device-name mysql-disk3 --zone=us-central1-c\n ```\n4. Ensure that ZFS utils packages are installed on your worker nodes. If it is not installed, ZFS packages can be installed.\n \n ```\n $ sudo su -\n $ sudo apt-get update\n $ apt-get install zfsutils-linux -y\n $ zpool import zfspv-pool\n ```\n \n Verify Zpool information.\n \n ```\n $ zpool list\n NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT\n zfspv-pool 14.5G 463M 14.0G - 0% 3% 1.0\n ```\n5. Verify the details of the ZFS dataset detail where the volume is present. This information is required in step 10. In the case of Statefulset, the ZFS dataset will be present on multiple nodes. So you should note down the details of the ZFS dataset and corresponding node information.\n \n ```\n $ sudo su\n $ root@gke-openebs-mysql-default-pool-dd23ce6b-tf5j:~# zfs list\n NAME USED AVAIL REFER MOUNTPOINT\n zfspv-pool 10.6G 3.42G 96K /zfspv-pool\n zfspv-pool/pvc-e299a9db-0903-417b-8034-03c3dc77af87 10.6G 13.6G 462M -\n ```\n\n From the above information, the ZFS dataset is present on Node `gke-openebs-mysql-default-pool-dd23ce6b-tf5j`. We have to update this node information in the nodeSelector field in step 10 and as `ownerNodeID` in step 11.\n\n6. Ensure OpenEBS ZFS driver pods are running in the `kube-system` namespace.\n \n ```\n $ kubectl get pods -n kube-system -l role=openebs-zfs\n NAME READY STATUS RESTARTS AGE\n openebs-zfs-controller-0 5/5 Running 0 91m\n openebs-zfs-node-29dlp 2/2 Running 0 14m\n openebs-zfs-node-bssq7 2/2 Running 0 14m\n openebs-zfs-node-p54tq 2/2 Running\t 0 14m\n ```\n7. Check the status of the application pod. It will be in the `Pending` state.\n \n ```\n $ kubectl get pod\n NAME READY STATUS RESTARTS AGE\n percona-9fbdb8678-z79vr 0/1 Pending 0 70m\n ```\n8. Get the PV details of the associated application\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO Delete Bound default/demo-vol1-claim openebs-zfspv 33m\n ```\n9. Create a directory and copy the YAML spec of all the associated PVs into it like below.\n \n ```\n $ mkdir mysql-restore\n $ cd mysql-restore/\n $ kubectl get pv pvc-e299a9db-0903-417b-8034-03c3dc77af87 -o yaml --export > pv1.yaml\n ```\n \n Note: If it is StatefulSet, take the YAML spec of all the associated PVs of that application.\n\n10. Modify the above-copied YAML with the new hostname in the copied YAML of PV. The following is a snippet of PV spec where it mentions the new node name where the ZFS volume is created. The node information will be obtained from step 5.\n ```\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: openebs.io/nodename\n operator: In\n values:\n - gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n persistentVolumeReclaimPolicy: Delete\n storageClassName: openebs-zfspv\n ```\n11. Also, update the node information where the ZFS dataset resides into `zv`(ZFS volume) cr. The node name has to be given to the path  `spec.ownerNodeID`.\n \n ```\n $ kubectl edit zv pvc-e299a9db-0903-417b-8034-03c3dc77af87 -n openebs\n ```\n \n The following is the snippet of the modified information.\n \n ```\n spec:\n capacity: \"10737418240\"\n compression: \"off\"\n dedup: \"off\"\n fsType: ext4\n ownerNodeID: gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n poolName: zfspv-pool\n ```\n12. Now get the PV and then delete the PV\n \n ```\n spec:\n capacity: \"10737418240\"\n compression: \"off\"\n dedup: \"off\"\n fsType: ext4\n ownerNodeID: gke-openebs-mysql-default-pool-dd23ce6b-tf5j\n poolName: zfspv-pool\n ```\n ```\n $ kubectl delete pv pvc-e299a9db-0903-417b-8034-03c3dc77af87\n persistentvolume \"pvc-e299a9db-0903-417b-8034-03c3dc77af87\" deleted\n ```\n \n The deletion of the PV will not be completed since it has the finaliser set with the PV. So we need to cancel the ongoing operation and then edit the PV and remove Finalizers. Once finalizers are removed, the volume will be automatically deleted.\n\n13. Verify that the PV of the application has been removed successfully.\n \n ```\n $ kubectl delete pv pvc-e299a9db-0903-417b-8034-03c3dc77af87\n persistentvolume \"pvc-e299a9db-0903-417b-8034-03c3dc77af87\" deleted\n ```\n14. Now, apply the updated YAML files of the PV. \n \n ```\n $ kubectl apply -f pv1.yaml \n ```\n \n Note: Perform the same for other PVs as well if the application is a StatefulSet.\n\n15. Verify that if PODs are started `Running` from `Pending` state.\n \n ```\n $ kubectl get pod -o wide\n NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\n percona-9fbdb8678-z79vr 1/1 Running 0 6h43m 10.4.2.2 gke-openebs-mysql-default-pool-dd23ce6b-tf5j \n ```\n \n Verify the PV, PVC, and ZV associated with the MySQL application.\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO Delete Bound default/demo-vol1-claim openebs-zfspv 150m\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-vol1-claim Bound pvc-e299a9db-0903-417b-8034-03c3dc77af87 10Gi RWO openebs-zfspv 7h7m\n\n $ kubectl get zv -n openebs\n NAME ZPOOL NODE SIZE STATUS FILESYSTEM AGE\n pvc-e299a9db-0903-417b-8034-03c3dc77af87 zfspv-pool gke-openebs-mysql-default-pool-dd23ce6b-tf5j 10737418240 Ready ext4 7h7m\n\n16. Login to the application and verify that you are able to access the data.\n ```\n $ kubectl exec -it percona-9fbdb8678-z79vr sh\n sh-4.2$ mysql -uroot -pk8sDem0;\n mysql: [Warning] Using a password on the command line interface can be insecure.\n Welcome to the MySQL monitor. Commands end with ; or \\g.\n Your MySQL connection id is 2\n Server version: 5.7.30-33 Percona Server (GPL), Release 33, Revision 6517692\n \n Copyright (c) 2009-2020 Percona LLC and/or its affiliates\n Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.\n \n Oracle is a registered trademark of Oracle Corporation and/or its\n affiliates. Other names may be trademarks of their respective\n owners.\n \n Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n \n mysql> SHOW DATABASES;\n +--------------------+\n | Database |\n +--------------------+\n | information_schema |\n | mysql |\n | performance_schema |\n | pets |\n | sys |\n +--------------------+\n 5 rows in set (0.11 sec)\n \n mysql> USE pets;\n Reading table information for completion of table and column names\n You can turn off this feature to get a quicker startup with -A\n \n Database changed\n mysql> SELECT * FROM cats;\n +----+---------+--------+------------+\n | id | name | owner | birth |\n +----+---------+--------+------------+\n | 1 | Sandy | Lennon | 2015-01-03 |\n | 2 | Cookie | Casey | 2013-11-13 |\n | 3 | Charlie | River | 2016-05-21 |\n +----+---------+--------+------------+\n 3 rows in set (0.00 sec)\n ```","notHasFeatureImage":false,"slug":"handling-node-down/cluster-scaling-on-zfs-localpv-backed-workloads"},{"id":24,"title":"Recover from Volume Multi Attach Error in On-Prem Kubernetes Clusters","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. OpenSource Enthusiast","date":"27-08-2020","tags":["Kubernetes"],"excerpt":"In this blog, we'll talk about recovering from volume multi attach error in On-Prem Kubernetes clusters.","content":"\nIf you have an unmanaged Kubernetes Cluster that you have deployed on-prem or on cloud, you would have noticed that your Stateful Application pods error out with Multi-attach error whenever the node running the stateful application is abruptly shut down.\n\nThis has been a long outstanding issue in Kubernetes and is being actively worked on. Please refer to the following Kubernetes issues:\n\n- [https://github.com/kubernetes/enhancements/pull/1116](https://github.com/kubernetes/enhancements/pull/1116)\n- [https://github.com/kubernetes/kubernetes/issues/86281](https://github.com/kubernetes/kubernetes/issues/86281)\n- [https://github.com/kubernetes/kubernetes/issues/53059#issuecomment-619428689](https://github.com/kubernetes/kubernetes/issues/53059#issuecomment-619428689)\n\nThe main reason for this issue being hard to resolve is that there is no right way to determine if the node is really shut down or if it is due to a network/split-brain condition to the master nodes. And it gets a little harder with Stateful applications as we need to really determine that data is written down from older nodes to the disks, before forcibly remounting onto the new node.\n\nIn this blog, I will provide an alternate approach that can be used to work around this issue and bring your applications back online. The solution is loosely based on the same approach that Managed Kubernetes clusters like GKE/EKS perform to handle this scenario. The managed clusters use out-of-band communication to determine if the node is shut down and delete the node resources.\n\nI will demonstrate the approach of removing the node resource as a safe way to recover volumes using the following example.\n\n## **Problem**\n\n1. ### Start with a Stateful application:\n I have a three node cluster with k8s version 1.15.3, to reproduce the Volume multi-attach error scenario. Deployed OpenEBS version 1.3, using cstor csi based volume and mounted to Percona pod scheduled in node csi-node2.mayalabs.io.\n ```\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node2.mayalabs.io Ready none 39d v1.15.3\n csi-node3.mayalabs.io Ready none 39d v1.15.3\n\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-csi-vol-claim Bound pvc-b39248ab-5a99-439b-ad6f-780aae30626c 10Gi RWO openebs-csi-cstor-sparse 72m\n\n\n $ kubectl get pods -owide\n NAME READY STATUS RESTARTS AGE NODE\n percona-6795d6fb68-pqvqh 1/1 Running 0 66m csi-node2.mayalabs.io\n ```\n\n VolumeAttachment resource created only for volume attached to Node2 in case of CSI based persistent volumes\n ```\n $ kubectl get volumeattachment\n NAME ATTACHER PV NODE ATTACHED AGE\n csi-9f7704015b456f146ce8c6c3bd80a5ec6cc55f4f5bfb90c61c250d0b050a283c openebs-csi.openebs.io pvc-b39248ab-5a99-439b-ad6f-780aae30626c csi-node2.mayalabs.io true 66m\n ```\n\n2. ### Node ShutDown:\n\n Shutting down kubelet service in Node2 as Percona application pod has been scheduled here, to make node `NotReady` state in the Kubernetes cluster.\n\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node2.mayalabs.io NotReady none 5m v1.15.3\n csi-node3.mayalabs.io Ready none 37d v1.15.3\n \n\n In this case, the Percona pod will get stuck in a container creating a state with a multi-attach error.\n\n## **Solution:**\n\nAlthough these solutions are generic to recover the volume from the multi attach error. Based on some extra steps, I have divided them into two different sanctions based on the type of Kubernetes volumes. One is dynamic in-tree or external Kubernetes volumes, an older way of provisioning Kubernetes volumes, and the other one is CSI based Kubernetes volumes.\n\n### **Dynamic/Static Provisioner Based Volumes:**\n\n#### **Deleting Node Resource:**\n\nHere after deleting the Node resource below node related events will be generated which will trigger the force deletion of the pods, and they apparently will be scheduled to other available nodes:\n\n $ kubectl delete nodes csi-node2.mayalabs.io\n node \"csi-node2.mayalabs.io\" deleted\n \n\n $ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n csi-master.mayalabs.io Ready master 39d v1.15.3\n csi-node1.mayalabs.io Ready none 39d v1.15.3\n csi-node3.mayalabs.io Ready none 37d v1.15.3\n \n\n##### **Check the kube-controller logs for events:**\n\n I1021 10:37:44.336523 1 attach_detach_controller.go:573] error removing node \"csi-node2.mayalabs.io\" from desired-state-of-world: failed to delete node \"csi-node2.mayalabs.io\" from list of nodes managed by attach/detach controller--the node still contains 1 volumes in its list of volumes to attach\n I1021 10:37:45.003243 1 event.go:258] Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"\", Name:\"csi-node2.mayalabs.io\", UID:\"30ca0f5e-3a8f-4d0f-99fc-776d7051fd3d\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'RemovingNode' Node csi-node2.mayalabs.io event: Removing Node csi-node2.mayalabs.io from Controller\n \n \n I1021 10:37:55.273070 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/openebs-csi-node-cjzlz\n I1021 10:37:55.318908 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/openebs-csi-node-cjzlz succeeded\n I1021 10:37:55.318979 1 gc_controller.go:62] PodGC is force deleting Pod: openebs/openebs-ndm-8ntsv\n I1021 10:37:55.352796 1 gc_controller.go:166] Forced deletion of orphaned Pod openebs/openebs-ndm-8ntsv succeeded\n I1021 10:37:55.354071 1 gc_controller.go:62] PodGC is force deleting Pod: openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\n I1021 10:37:55.420779 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"openebs\", Name:\"cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\", UID:\"\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'TaintManagerEviction' Cancelling deletion of Pod openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn\n I1021 10:37:55.442403 1 gc_controller.go:166] Forced deletion of orphaned Pod openebs/cspc-sparse-disk-pool-gg82-d9b4bff4d-9fmbn succeeded\n I1021 10:37:55.442568 1 gc_controller.go:62] PodGC is force deleting Pod: default/percona-6795d6fb68-b7dvl\n I1021 10:37:55.446368 1 event.go:258] Event(v1.ObjectReference{Kind:\"ReplicaSet\", Namespace:\"openebs\", Name:\"cspc-sparse-disk-pool-gg82-d9b4bff4d\", UID:\"04f87ed3-c401-4688-8691-0716dc4693fe\", APIVersion:\"apps/v1\", ResourceVersion:\"7063677\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: cspc-sparse-disk-pool-gg82-d9b4bff4d-q2l2b\n I1021 10:37:55.541929 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-b7dvl\", UID:\"\", APIVersion:\"\", ResourceVersion:\"\", FieldPath:\"\"}): type: 'Normal' reason: 'TaintManagerEviction' Cancelling deletion of Pod default/percona-6795d6fb68-b7dvl\n I1021 10:37:55.599155 1 gc_controller.go:166] Forced deletion of orphaned Pod default/percona-6795d6fb68-b7dvl succeeded\n I1021 10:37:55.599224 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/kube-proxy-b9q25\n I1021 10:37:55.613517 1 event.go:258] Event(v1.ObjectReference{Kind:\"ReplicaSet\", Namespace:\"default\", Name:\"percona-6795d6fb68\", UID:\"50b82272-9874-4688-8362-7c759ae63aef\", APIVersion:\"apps/v1\", ResourceVersion:\"7063669\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: percona-6795d6fb68-pqvqh\n W1021 10:37:55.621461 1 reconciler.go:328] Multi-Attach error for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" Volume is already exclusively attached to node csi-node2.mayalabs.io and can't be attached to another\n I1021 10:37:55.629191 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-pqvqh\", UID:\"26ee6109-9d7e-4729-843b-18bb88926c87\", APIVersion:\"v1\", ResourceVersion:\"7063944\", FieldPath:\"\"}): type: 'Warning' reason: 'FailedAttachVolume' Multi-Attach error for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" Volume is already exclusively attached to one node and can't be attached to another\n I1021 10:37:55.742498 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/kube-proxy-b9q25 succeeded\n I1021 10:37:55.742697 1 gc_controller.go:62] PodGC is force deleting Pod: kube-system/calico-node-4fv7n\n I1021 10:37:55.787435 1 gc_controller.go:166] Forced deletion of orphaned Pod kube-system/calico-node-4fv7n succeeded\n \n\n### **CSI Based Volumes:**\n\n#### **Attach-Detach Controller:**\n\nIf the volume is created using CSI Provisioner, a custom resource `volumeattachment` would be created. The Attach-detach controller will wait for the `maxWaitForUnmountDuration` i.e., 6 minutes to forcefully detach the attached volume from the node. Then the CR will be recreated and attach to any available node. To recover from the multi-attach error, this `volumeattachment` CR can be deleted along with node CR. Therefore the time taken to mount the volume on a new node will be reduced by 6 minutes.\n\n $ kubectl get volumeattachment pvc-b39248ab-5a99-439b-ad6f-780aae30626c\n NAME ATTACHER PV NODE ATTACHED AGE\n csi-6ae3ead0d1c3a6e73e7d4c8e27f9098b927f3e4edc21bcf6bb7cf3fcdb4101de openebs-csi.openebs.io pvc-b39248ab-5a99-439b-ad6f-780aae30626c csi-node1.mayalabs.io true 11m\n \n\n \n $ kubectl logs -f kube-controller-manager-csi-master.mayalabs.io -n kube-system\n\n \n ```\n W1021 10:43:55.629673 1 reconciler.go:232] attacherDetacher.DetachVolume started for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") on node \"csi-node2.mayalabs.io\" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching\n I1021 10:43:55.671000 1 operation_generator.go:526] DetachVolume.Detach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") on node \"csi-node2.mayalabs.io\" \n I1021 10:43:55.730390 1 reconciler.go:288] attacherDetacher.AttachVolume started for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" \n I1021 10:43:55.752273 1 operation_generator.go:358] AttachVolume.Attach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" (UniqueName: \"kubernetes.io/csi/openebs-csi.openebs.io^pvc-b39248ab-5a99-439b-ad6f-780aae30626c\") from node \"csi-node1.mayalabs.io\" \n I1021 10:43:55.753344 1 event.go:258] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"percona-6795d6fb68-pqvqh\", UID:\"26ee6109-9d7e-4729-843b-18bb88926c87\", APIVersion:\"v1\", ResourceVersion:\"7063944\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulAttachVolume' AttachVolume.Attach succeeded for volume \"pvc-b39248ab-5a99-439b-ad6f-780aae30626c\" \n ```\n \n\nThat's it for today's post. I hope you find it helpful. Feedback and comments are appreciated.\n","notHasFeatureImage":false,"slug":"recover-from-volume-multi-attach-error-in-on-prem-kubernetes-clusters"},{"id":25,"title":"Resize Kubernetes StatefulSets without impact","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, Passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"25-08-2020","tags":["Kubernetes","OpenEBS"],"excerpt":"Read this post if you are a cStor CSI user who's looking to resize statefulsets without any impact of StatefulSet applcations. ","content":"\nIn large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. MayaData, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. cStor is one of the storage engines of OpenEBS.\n\nThis blog is for OpenEBS users, specifically cStor CSI users looking to resize their ***Kubernetes StatefulSets*** without any impact of StatefulSet applications.\n\n### **Steps involved to resize** ***Kubernetes StatefulSets***\n\n1. Increase the volume size of the PVC.\n\n2. Update StatefulSet volumeClaimTemplate storage capacity.\n\n### Infrastructure details:\n\n Kubernetes Cluster: Bare Metal\n Kubernetes Version: v1.17.2\n OpenEBS Version: 2.0.0\n\n### Installation of CStor setup:\n\nApplied OpenEBS 2.0.0 version of cStor operator [yaml](https://github.com/openebs/charts/blob/gh-pages/2.0.0/cstor-operator.yaml) and ndm operator [yaml](https://github.com/openebs/charts/blob/gh-pages/2.0.0/ndm-operator.yaml) via kubectl apply and provisioned cStor pools using CSPC API by following the steps mentioned [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md).\n\n system@master$ kubectl get cspc -n openebs\n NAME HEALTHYINSTANCES PROVISIONEDINSTANCES DESIREDINSTANCES AGE\n cspc-stripe 3 3 3 3m23s\n\n### Create StorageClass:\n\nCreate a StorageClass pointing to the above cStor pool(cspc-stripe) which will help in provisioning the cStor volume. For expanding the volumes **allowVolumeExpansion** parameter should set to true\n\n system@master$ cat csi-sc.yaml \n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-csi-sc\n provisioner: cstor.csi.openebs.io\n allowVolumeExpansion: true\n parameters:\n cas-type: cstor\n replicaCount: \"1\"\n cstorPoolCluster: cspc-stripe\n\n### Provision StatefulSet:\n\nCreate StatefulSet to point to the above storageclass. In this example, mongo statefulset will be provisioned and volume size will be expanded from 5Gi to 15Gi. Below is the sample statefulset used for expansion:\n\n system@master$ cat mongo-sts.yaml \n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: mongo\n Namespace: mongo-ns\n spec:\n selector:\n matchLabels:\n role: mongo\n environment: test\n serviceName: \"mongo\"\n replicas: 3\n template:\n metadata:\n labels:\n role: mongo\n environment: test\n spec:\n terminationGracePeriodSeconds: 10\n containers:\n - name: mongo\n image: mongo:latest\n imagePullPolicy: IfNotPresent\n command:\n - mongod\n - \"--replSet\"\n - rs0\n ports:\n - containerPort: 27017\n volumeMounts:\n - name: mongo-persistent-storage\n mountPath: /data/db\n - name: mongo-sidecar\n image: cvallance/mongo-k8s-sidecar\n ...\n ...\n volumeClaimTemplates:\n - metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: \"openebs-csi-sc\"\n accessModes: [\"ReadWriteOnce\"]\n resources:\n requests:\n storage: 5Gi\n\nAfter applying the above YAML. This results in the provisioning of volumes with **5Gi** capacity as mentioned in **volumeClaimTemplates,** and applications are in Running state.\n\n system@master$ kubectl get pods -n mongo-ns\n NAME READY STATUS RESTARTS AGE\n mongo-0 2/2 Running 0 8m24s\n mongo-1 2/2 Running 0 7m5s\n mongo-2 2/2 Running 0 6m4s\n \n -----------------------------------------------------------------------------------\n system@master$ kubectl get pvc -n mongo-ns\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n mongo-persistent-storage-mongo-0 Bound pvc-4926e6bb-3226-4de9-add5-e603ea2b948b 5Gi RWO openebs-csi-sc 6m48s\n mongo-persistent-storage-mongo-1 Bound pvc-fcde1fde-65ee-48ab-8f79-b7c68edfb934 5Gi RWO openebs-csi-sc 5m29s\n mongo-persistent-storage-mongo-2 Bound pvc-d94e5304-7f29-4d6c-b157-1e65f75511f1 5Gi RWO openebs-csi-sc 4m28s\n\nNow verify the size of the volume by exec into any one of the application pods. In below output **/dev/sde** is the persistent volume and it’s capacity is **5G**.\n\n system@master$ kubectl exec -it mongo-0 -n mongo-ns -c mongo -- df -h\n Filesystem Size Used Avail Use% Mounted on\n overlay 98G 12G 82G 13% /\n tmpfs 64M 0 64M 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sde 4.8G 311M 4.5G 7% /data/db\n /dev/sda1 98G 12G 82G 13% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /proc/acpi\n tmpfs 3.9G 0 3.9G 0% /proc/scsi\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nNow, let’s resize the volume capacity by following the steps mentioned in the blog's beginning.\n\nStep1: Increase the volume size of PVC\n\nExpand the size of the PVC size by applying below command on all the StatefulSet volumes:\n\n kubectl patch pvc mongo-persistent-storage-mongo-0 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n \n kubectl patch pvc mongo-persistent-storage-mongo-1 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n \n \n kubectl patch pvc mongo-persistent-storage-mongo-2 -p '{ \"spec\": { \"resources\": { \"requests\": { \"storage\": \"15Gi\" }}}}' -n mongo-ns\n\nAfter patching the above PVCs **openebs-cstor-csi** plugin is responsible for performing the resize operation on cStor volume. Verify whether volumes are expanded successfully by performing the following commands\n\n system@master: kubectl get pvc -n mongo-ns\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n mongo-persistent-storage-mongo-0 Bound pvc-4926e6bb-3226-4de9-add5-e603ea2b948b 15Gi RWO openebs-csi-sc 32m\n mongo-persistent-storage-mongo-1 Bound pvc-fcde1fde-65ee-48ab-8f79-b7c68edfb934 15Gi RWO openebs-csi-sc 30m\n mongo-persistent-storage-mongo-2 Bound pvc-d94e5304-7f29-4d6c-b157-1e65f75511f1 15Gi RWO openebs-csi-sc 29m\n\nAbove PVC shows that volumes are expanded successfully. Now verify capacity by exec in to any one of the application pods\n\n system@master$ kubectl exec -it mongo-0 -n mongo-ns -c mongo -- df -h\n Filesystem Size Used Avail Use% Mounted on\n overlay 98G 12G 82G 13% /\n tmpfs 64M 0 64M 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sde 15G 313M 15G 3% /data/db\n /dev/sda1 98G 12G 82G 13% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /proc/acpi\n tmpfs 3.9G 0 3.9G 0% /proc/scsi\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nNote: This step only helps to resize the volumes that are already provisioned for consuming via statefulsets. If StatefulSet is scaled up, then the newly provisioning volume will have the old size(since the volumeClaimTemplate is not yet updated in the application spec).\n\n**Step2: Update StatefulSet volumeClaimTemplate storage size**\n\nSince natively resizing of statefulset volumeClaimTemplates is not yet supported in Kubernetes, one needs to follow this step to update size in the volume claim template. For further updates, track the enhancement proposal [here](https://github.com/kubernetes/enhancements/pull/1848).\n\nUpdate the capacity in volumeClaimTemplate of applied statefulset yaml lke below example:\n\n system@master$ cat mongo-sts.yaml\n cat mongo-sts.yaml \n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: mongo\n namespace: mongo-ns\n spec:\n selector:\n matchLabels:\n role: mongo\n environment: test\n serviceName: \"mongo\"\n replicas: 3\n ...\n ...\n volumeClaimTemplates:\n - metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: \"openebs-csi-sc\"\n accessModes: [\"ReadWriteOnce\"]\n resources:\n requests:\n storage: 15Gi\n\nNow delete the statefulset without deleting the statefulset pods for not to have any down time for application and apply the updated volumeClaimTemplate.\n\n system@master$ kubectl delete sts mongo -n mongo-ns --cascade=false\n statefulset.apps \"mongo\" deleted\n \n system@master$ kubectl apply -f mongo-sts.yaml \n statefulset.apps/mongo created\n service/mongo unchanged\n\nVerify changes by describing the sts\n\n system@master$ kubectl describe sts mongo -n mongo-ns\n \n Name: mongo\n Namespace: mongo-ns\n CreationTimestamp: Mon, 17 Aug 2020 16:38:51 +0530\n Selector: environment=test,role=mongo\n ...\n ...\n Volume Claims:\n Name: mongo-persistent-storage\n StorageClass: openebs-csi-sc\n Labels: \n Annotations: \n Capacity: 15Gi\n Access Modes: [ReadWriteOnce]\n\nIn the above output, capacity has been updated from **5Gi** to **15Gi**. So successfully updated the statefulset capacity without any down time.\n","notHasFeatureImage":false,"slug":"resize-kubernetes-statefulsets-without-impact"},{"id":26,"title":"Handling node down/cluster scaling on LocalPV backed workloads","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"21-08-2020","tags":["LocalPV","OpenEBS"],"excerpt":"In this article, we'll discuss the steps that is getting handled when a node down/cluster scaling on LocalPV backs workloads.","content":"\nKubernetes is increasingly used for running production-grade stateful services. Organizations are making progress on a containerized form of their production workloads for running in Kubernetes. There are already solutions available for the containerized version of stateful applications, network, storage, etc.\n\n### Handling node down / cluster scaling on Local PV backed workloads\n\nOpenEBS is one of the leading containerized storage solutions for Kubernetes, and it is a rapidly growing Sandbox project in CNCF. MayaData is the primary maintainer and contributor of OpenEBS along with other companies. MayaData also contributed another open source project, Litmus, into CNCF, which does mostly Chaos engineering in Kubernetes, which helps SREs and developers to do all kinds of testing of their applications and components in Kubernetes before going into production.\n\nIt is a must requirement of a persistent storage solution for running the stateful application, be it a **Deployment** or **StatefulSet**. OpenEBS provides many storage engines, and each storage engine is suitable for specific applications or workloads. Some engines provide storage level synchronous replication, capable of taking snapshots and cloning, backup and restore, volume expansion, CSI complaint, performance-oriented, etc. So choosing the engine based on the workload requirement is an important activity.\n\nOpenEBS provides dynamic provisioning of LocalPV using an external device, and this external device will be allocated entirely to an application. You can also use the partitioned disk for using OpenEBS LocalPV by using the `openebs-hostpath` storage engine. In this article, we provisioned a MySQL deployment on an OpenEBS LocalPV device dynamically.\n\nThis article is a step-by-step instruction. We will mention how a MySQL application deployment running on ***OpenEBS LocalPV*** device volume is getting handled when a Node down scenario or a cluster scale down situation happens in the GKE cluster. In GKE and some other managed clusters like EKS, the node name will change if the cluster undergoes a scale down and scale-up operation has performed. So the application running on the ***OpenEBS LocalPV*** device will not be able to attach to the new node since the corresponding PV has volume node affinity. We need to update the new node name details in PV, where the disk got attached.\n\nIn this article, we discuss the steps that need to be performed to make the application into a running state when a Node down / scale down cluster scenario has happened. This situation is usually required in case of managed clusters where the node name will get changed during this scenario. As stated earlier, the following approach works fine for both Deployment type and StatefulSet type, but ensure that the following steps are correctly satisfied. Let’s start with the step by step instructions once you have scaled up the cluster after a scale down scenario.\n\n1. Verify all nodes are now in Ready state.\n ```\n $ kubectl get node\n NAME STATUS ROLES AGE VERSION\n gke-openebs-mysql-default-pool-d55297a7-bjjp Ready 74s v1.16.13-gke.1\n gke-openebs-mysql-default-pool-d55297a7-j1vm Ready 80s v1.16.13-gke.1\n gke-openebs-mysql-default-pool-d55297a7-pvg4 Ready 85s v1.16.13-gke.1\n ```\n2. Ensure OpenEBS pods are in Running state.\n ```\n $ kubectl get pod -n openebs\n NAME READY STATUS RESTARTS AGE\n maya-apiserver-76cb4df9b8-wpbf6 1/1 Running 0 22m\n openebs-admission-server-5cf696b8d5-d97bn 1/1 Running 0 22m\n openebs-localpv-provisioner-7654f6dbd9-hskq8 1/1 Running 0 22m\n openebs-ndm-7dtts 1/1 Running 0 2m19s\n openebs-ndm-c4r4m 1/1 Running 0 2m23s\n openebs-ndm-lnb5c 1/1 Running 0 2m12s\n openebs-ndm-operator-6cfc59b69b-684nx 1/1 Running 0 22m\n openebs-provisioner-7d9884d4ff-tfcxj 1/1 Running 0 22m\n openebs-snapshot-operator-7ff577c889-kfttj 2/2 Running 0 22m\n ```\n3. Check the status of the application pod. It will be in the `Pending` state.\n ```\n $ kubectl get pod\n NAME READY STATUS RESTARTS AGE\n percona-9fbdb8678-lncd5 0/1 Pending 0 17m\n ```\n4. Label all the nodes with the same custom label used in the `nodeSelector` field in the STS app. In my case, there is no custom node label used in application deployment. So we are skipping this step.\n\n5. Attach the disk randomly to any node in the same zone. Note down the device name and node name where it is getting attached. This information will be needed in step 9.\n ```\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-bjjp --disk mysql-disk1 --device-name mysql-disk1 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-j1vm --disk mysql-disk2 --device-name mysql-disk2 --zone=us-central1-c\n\n $ gcloud compute instances attach-disk gke-openebs-mysql-default-pool-d55297a7-pvg4 --disk mysql-disk3 --device-name mysql-disk3 --zone=us-central1-c\n ```\n6. Verify BDs are updated with new node names\n ```\n $ kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n blockdevice-4f51859193d333687a873af7acf8ad78 gke-openebs-mysql-default-pool-d55297a7-j1vm 32212254720 Unclaimed Active 37m\n blockdevice-967d7816c2a2d73b91c8c6310dc70465 gke-openebs-mysql-default-pool-d55297a7-bjjp 32212254720 Claimed Active 37m\n blockdevice-ddfc782ea661fc9007a896438f483e3d gke-openebs-mysql-default-pool-d55297a7-pvg4 32212254720 Unclaimed Active 37m\n ```\n7. Get the PV details of the associated application\n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 5G RWO Delete Bound default/demo-vol1-claim openebs-device 33m\n ```\n8. Create a directory and copy the YAML spec of all the associated PVs into it like below\n ```\n $ mkdir mysql-restore\n $ cd mysql-restore/\n $ kubectl get pv pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 -o yaml --export > pv1.yaml\n ```\n Note: If it is StatefulSet, take the YAML spec of all the associated PVs of that application.\n\n9. Modify the above-copied YAML with the new hostname in the copied YAML of PV. The following is that snippet of PV spec where it mentions the new node name where the Local disk is attached.\n ```\n path: /dev/disk/by-id/scsi-0Google_PersistentDisk_mysql-disk1\n nodeAffinity:\n required:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - gke-openebs-mysql-default-pool-d55297a7-bjjp\n ```\n10. Now get the PV and then delete the PV\n \n ```\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999 5G RWO Delete Bound default/demo-vol1-claim openebs-device 36m\n \n $ kubectl delete pv pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999\n persistentvolume \"pvc-5cd17649-efe4-46e1-a5f3-f779b0e03999\" deleted\n ```\n The deletion of the PV will not be completed since it has the finaliser set with the PV. So we need to cancel the ongoing operation and then edit the PV and remove Finalizers. Once finalizers are removed, the volume will be automatically deleted.\n\n11. Verify that the PV of the application has been removed successfully.\n ```\n $ kubectl get pv\n No resources were found in the default namespace.\n ```\n12. Now, apply the updated YAML files of the PV.\n ```\n $ kubectl apply -f pv1.yaml \n ```\n Note: Perform the same for other PVs as well if the application is a StatefulSet.\n\n13. Verify that if PODs are started `Running` from `Pending` state.\n ```\n $ kubectl get pod -o wide\n NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES\n percona-9fbdb8678-lncd5 1/1 Running 0 29m 10.16.0.2 gke-openebs-mysql-default-pool-d55297a7-bjjp \n ```\n14. Log in to the application and verify that you are able to access the data.\n\n ```\n $ kubectl exec -it percona-9fbdb8678-lncd5 sh\n kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.\n sh-4.2$ mysql -uroot -pk8sDem0;\n mysql: [Warning] Using a password on the command line interface can be insecure.\n Welcome to the MySQL monitor. Commands end with ; or \\g.\n Your MySQL connection id is 2\n Server version: 5.7.30-33 Percona Server (GPL), Release 33, Revision 6517692\n \n Copyright (c) 2009-2020 Percona LLC and/or its affiliates\n Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.\n \n Oracle is a registered trademark of Oracle Corporation and/or its\n affiliates. Other names may be trademarks of their respective\n owners.\n \n Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\n mysql> SHOW DATABASES;\n +--------------------+\n | Database |\n +--------------------+\n | information_schema |\n | mysql |\n | performance_schema |\n | pets |\n | sys |\n +--------------------+\n 5 rows in set (0.07 sec)\n \n mysql> use pets;\n \n Reading table information for completion of table and column names\n You can turn off this feature to get a quicker startup with -A\n \n Database changed\n mysql> SELECT * FROM cats;\n +----+---------+--------+------------+\n | id | name | owner | birth |\n +----+---------+--------+------------+\n | 1 | Sandy | Lennon | 2015-01-03 |\n | 2 | Cookie | Casey | 2013-11-13 |\n | 3 | Charlie | River | 2016-05-21 |\n +----+---------+--------+------------+\n 3 rows in set (0.00 sec)\n ```","notHasFeatureImage":false,"slug":"handling-node-down/cluster-scaling-on-localpv-backed-workloads"},{"id":27,"title":"OpenEBS-Velero backup/restore of stateful applications","author":"Vishnu Itta","author_info":"Developer who is always eager to learn, loves math, algorithms, & programming. Have a good experience in storage protocols, ZFS, FreeBSD internals, Linux, device drivers.","date":"04-08-2020","tags":["OpenEBS","Velero"],"excerpt":"In this blog, we will go through the backup/restore solution provided to its users by OpenEBS with the help of Velero.","content":"\n### **Backup/Restore of stateful applications in OpenEBS**\n\nSREs play a crucial role in automating operations. This role includes handling infrastructure upgrades and the upgrades of software running on that infrastructure. When running stateful workloads, the data must be backed up before any upgrades.\n\nAdditionally, workloads on Kubernetes are often different than traditional monolithic applications since Kubernetes supports microservices and loosely coupled workloads. This brings-in the need for a `cloud-native` design of backup/restore per workload or application. Once implemented this approach provides benefits to users such as an app-centric view, ease of management, setting of RPO/RTO at the workload level, and others. To learn more about cloud native backup drivers and requirements - please read this [thenewstack.io article](https://thenewstack.io/cloud-native-backups-disaster-recovery-and-migrations-on-kubernetes/).\n\nThe below graphic shows microservices, and their loosely coupled nature. Managing backups and the tuning of RPO/RTO per microservice is required to preserve the loosely coupled nature of these workloads and of the teams responsible for them.\n![OpenEBS Velero backup/restore of stateful applications: fig 1](https://lh4.googleusercontent.com/JyIxZZtj-1rBOGo2z1bKcYlD9-halM8dXpChtOIPro_aZEhQObTL_K5_Be_hLgqXl7aT68jYqFjNE9C6rZK0IRaV_neH4DURZhUr3z5FkVirzMirk_z8FiscY6_sb3JPhlAx1XRL)\nThe following are **challenges** in redesigning backup/restore of K8s applications:\n\n- Ability to look from the application view rather than only the data\n- Dependencies with K8s constructs like node names, PVCs, storage classes, topologies in them\n- Snapshots of selected volumes rather than entire mount point/disk in hyper-converged infrastructure\n- K8s application deployment workflow to copy data during a restore whether using operators or another method\n\nIn this blog, we will go through a backup/restore solution for ***OpenEBS*** with the help of ***Velero***. OpenEBS is leading open source container attached storage solution, and Velero is one of the most used open-source tools for backups in Kubernetes environments.\n\n### **Backup/Restore solution**\n\nThis solution, with Velero, provides a declarative way for users to specify\n\n- object storage as the destination location to store backed-up data\n- K8s applications and its resources which are part of the backup\n- storage provider configuration to backup/restore data\n- schedules to take a backup at regular intervals\n- restore selected resources from the selected backup\n- app-specific hooks to execute before and after performing backup/restore\n\nLet's start by examining what OpenEBS, as container attached storage, offers. It has many kinds of storage engines as below:\n\n- Local PV host path and block device\n- Jiva\n- CStor\n- Local PV ZFS\n\n### **Local PV / Jiva Volumes + Restic + Restore Item Action plugin**\n\nOpenEBS Local PV volumes are prominent among cloud-native applications. These applications themselves take care of replicating data and snapshotting. As the name suggests, these are used in hyper-converged infrastructure. This brings in a few challenges during backup/restoration.\n\nLocal PVs related storage classes will have `WaitForFirstConsumer`. It provides delayed binding of volumes until Pod gets scheduled on the node. K8s sets the node name as an annotation on PVC. Restoring that PVC onto the destination cluster will leave the pod in pending state. This solution takes care of it by applying node name mapping in PVC annotation. Restore Item Action plugin, contributed by OpenEBS, in Velero performs this mapping.\n\nVelero's in-built file-based `Restic` plugin helps in backing up the application's data. `Init` container will be injected into the application to copy data from the destination location to the volume.\n\nJiva volumes are used by applications that need replication functionality from storage providers. File-based `Restic` plugin can be used to backup/restore applications using jiva volumes.\n\n### **CStor / Local PV ZFS + Velero plugin**\n\nStandard applications use OpenEBS cStor volumes. These applications need replication, snapshotting, cloning, etc., from storage providers. Also, these applications are free to run-anywhere-in-the-cluster nature. OpenEBS have a cStor velero-plugin for crash-consistent backup of data. This plugin performs pausing of IOs on the volume before the snapshot is taken on all the replicas. It resumes IOs once the point-in-time snapshot is taken. It reads the snapshot content from one of the replicas and backs it up at the object location. When a Schedule is newly created, full data will be backed up at the destination location. The plugin manages the completion status of the backup. On the next iterations, the plugin just backups the changes from previous successful backup.\n\nBased on the replication settings of cStor at the destination, the plugin takes care of copying data to the configured number of replicas. The connection between iSCSI target pod and replica pods is not established until data is copied into the volume. This avoids usage of the PV by any application during restore time.\n\nCStor plugin also provides an option to keep backups locally with the main volume in the form of local snapshots. Users can create a backup of applications and with local snapshots. Restoring these backups, which consists of local snapshots, can be done in a different namespace of the same cluster.\n\nWork-related to local PV ZFS plugin is in-active development to provide all the features as mentioned above and much more.\n\n### **Summary**\n\nTo summarize, a new kind of thinking (or) a new kind of approach is required to do cloud-native backups. OpenEBS has made tremendous progress in achieving it and provides various features and flavors of backup/restoration of its volumes.\n\n- File-based\n- Point-in-time block-based local snapshot and restore\n- Full and incremental block-based remote backup and restore\n- Per-workload backups\n- K8s resources transformations\n\nCredits to the Velero team for building a wonderful open-source backup/restore tool.\n\nPlease visit [https://openebs.io/docs](/docs), [https://velero.io](https://velero.io/) for more details about storage class, backup/restore examples, Velero.\n","notHasFeatureImage":false,"slug":"openebs-velero-backup/restore-of-stateful-applications"},{"id":28,"title":"Kubernetes StatefulSet on ppc64le using OpenEBS LocalPV provisioner","author":"Peeyush Gupta","author_info":"Peeyush, Sr. Developer Advocate, DigitalOcean, is a cloud enthusiast with 5+ years of experience in developing cloud platforms and helping customers migrate their legacy applications to the cloud.","date":"16-07-2020","tags":["OpenEBS","Kubernetes","LocalPV"],"excerpt":"In this blog, we'll explain how to install OpenEBS on Kubernetes StatefulSet running on the ppc64le platform & to using the OpenEBS LocalPV provisioner to deploy a StatefulSet.","content":"\nGuest post by Peeyush Gupta, Sr. Developer Advocate, DigitalOcean\n\n**OpenEBS** is the leading open-source project for container-attached and container-native storage on **Kubernetes**. OpenEBS adopts Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller. OpenEBS implements granular storage policies and isolation that enable users to optimize storage for each specific workload. OpenEBS runs in userspace and does not have any Linux kernel module dependencies. Here is how the setup looks like for OpenEBS LocalPV hostpath:\n![Kubernetes StatefulSet on ppc64le using OpenEBS LocalPV provisioner](https://lh4.googleusercontent.com/-erccwTcJCmyJGswEZ3Pul1-pvJO-kvn34nr22mqYumR1IHUVhX8BWOeennt1u91EYUKtpUfAPBSiP1XD_1z6XYmG8Tlywvl9GellLpkr8EyYTFLXT3YpIZ_nneRcen_G8uKVV6Q)\nIn this tutorial, we will see how we can install OpenEBS on Kubernetes running on the ppc64le platform. Then we will use the OpenEBS LocalPV provisioner to deploy a StatefulSet. The ppc64le servers used in this tutorial are running on IBM Cloud.\nKubernetes support multi-arch platforms. You can deploy the cluster on ppc64le based servers using kubeadm. For this tutorial, I am using a 3 node cluster:\n\n root@openebs-k8s-server:~# kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n openebs-k8s-server Ready master 17h v1.18.5\n openebs-worker-1 Ready 17h v1.18.5\n openebs-worker-2 Ready 17h v1.18.5\n\nWe will be deploying OpenEBS lite on this cluster. Here is an excellent blog post on how to do that: [https://help.mayadata.io/hc/en-us/articles/360031969532-Installing-OpenEBS-with-only-Local-PV-support](https://help.mayadata.io/hc/en-us/articles/360031969532-Installing-OpenEBS-with-only-Local-PV-support). To get started, deploy the openebs-operator-lite.yaml file for ppc64le using:\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-lite-ppc64le.yaml\n\nOnce deployed, you should be able to see the node-disk-manager and LocalPV components running. Note that node-disk-manager is not a mandatory component for provisioning LocalPV hostpath volumes. OpenEBS LocalPV provisioner can be installed and run standalone as well.\n\n root@openebs-k8s-server:~# kubectl get pods --all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n kube-system coredns-66bff467f8-njjc9 1/1 Running 0 17h\n kube-system coredns-66bff467f8-tndsx 1/1 Running 0 17h\n kube-system etcd-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-apiserver-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-controller-manager-openebs-k8s-server 1/1 Running 0 17h\n kube-system kube-proxy-55fbj 1/1 Running 0 17h\n kube-system kube-proxy-gt5rw 1/1 Running 0 17h\n kube-system kube-proxy-l5pz2 1/1 Running 0 17h\n kube-system kube-scheduler-openebs-k8s-server 1/1 Running 0 17h\n kube-system weave-net-c2gmk 2/2 Running 1 17h\n kube-system weave-net-qp5c7 2/2 Running 0 17h\n kube-system weave-net-trgr6 2/2 Running 1 17h\n openebs openebs-localpv-provisioner-7bb7dc9958-ln284 1/1 Running 0 16h\n openebs openebs-ndm-kqhpr 1/1 Running 0 16h\n openebs openebs-ndm-nxswk 1/1 Running 0 16h\n openebs openebs-ndm-operator-746d6cd4fd-bm2fp 1/1 Running 1 16h\n\nNext, we will deploy the storage class. This storage class will be used with PersistentVolumeClaim to create volumes.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-lite-sc.yaml \n\nThis will create 2 storage classes, openebs-device, and openebs-hostpath, on the cluster. \n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get sc\n NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\n openebs-device openebs.io/local Delete WaitForFirstConsumer false 6m12s\n openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 6m12s\n\nIn our case, we are interested in the `openebs-hostpath` storage class for this tutorial. Now, we are ready to deploy the StatefulSet that will consume the volume created using the above storage class. Here is a sample StatefulSet:\n\n apiVersion: apps/v1\n kind: StatefulSet\n metadata:\n name: local-test\n spec:\n serviceName: \"local-service\"\n replicas: 1\n selector:\n matchLabels:\n app: local-test\n template:\n metadata:\n labels:\n app: local-test\n spec:\n containers:\n - name: test-container\n image: busybox\n command:\n - \"/bin/sh\"\n args:\n - \"-c\"\n - \"sleep 100000\"\n volumeMounts:\n - name: openebs-localpv-hostpath\n mountPath: /usr/test-pod\n volumeClaimTemplates:\n - metadata:\n name: openebs-localpv-hostpath\n spec:\n accessModes: [ \"ReadWriteOnce\" ]\n storageClassName: \"openebs-hostpath\"\n resources:\n requests:\n storage: 5G\n\nThe above YAML creates a StatefulSet named `local-test`, which has a container named `test-container`. This container has a volume mounted at path `/usr/test-pod`. The claim for this particular volume references the `openebs-hostpath` storage class. We will save this YAML using the name openebs-localpv-stateful.yaml, and this can be deployed using kubectl create:\n\n kubectl apply -f openebs-localpv-stateful.yaml\n\nYou can verify the respective pv and pvc:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n openebs-localpv-hostpath-local-test-0 Bound pvc-e61a4156-b0bb-4199-b991-9c42b1830ec5 5G RWO openebs-hostpath 17s\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-e61a4156-b0bb-4199-b991-9c42b1830ec5 5G RWO Delete Bound default/openebs-localpv-hostpath-local-test-0 openebs-hostpath 18s\n\nOnce the volume is bound to the claim, the relevant pod i.e. local-test-0 comes up in running state:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl get pods --all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n default local-test-0 1/1 Running 0 13s\n kube-system coredns-66bff467f8-njjc9 1/1 Running 0 21h\n kube-system coredns-66bff467f8-tndsx 1/1 Running 0 21h\n kube-system etcd-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-apiserver-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-controller-manager-openebs-k8s-server 1/1 Running 0 21h\n kube-system kube-proxy-55fbj 1/1 Running 0 21h\n kube-system kube-proxy-gt5rw 1/1 Running 0 21h\n kube-system kube-proxy-l5pz2 1/1 Running 0 21h\n kube-system kube-scheduler-openebs-k8s-server 1/1 Running 0 21h\n kube-system weave-net-c2gmk 2/2 Running 1 21h\n kube-system weave-net-qp5c7 2/2 Running 0 21h\n kube-system weave-net-trgr6 2/2 Running 1 21h\n openebs openebs-localpv-provisioner-6fc9664557-njvbg 1/1 Running 0 24m\n openebs openebs-ndm-754qs 1/1 Running 0 24m\n openebs openebs-ndm-operator-798f74c6b9-24jvv 1/1 Running 0 24m\n openebs openebs-ndm-x96ks 1/1 Running 0 24m\n\nIf you describe the pvc, you can see the volume is being provisioned using openebs-localpv-provsioner:\n\n root@openebs-k8s-server:~/openebs-localpv# kubectl describe pvc\n Name: openebs-localpv-hostpath-local-test-0\n Namespace: default\n StorageClass: openebs-hostpath\n Status: Bound\n Volume: pvc-b89a1c70-beeb-4bdc-a555-348b01832443\n Labels: app=local-test\n Annotations: pv.kubernetes.io/bind-completed: yes\n pv.kubernetes.io/bound-by-controller: yes\n volume.beta.kubernetes.io/storage-provisioner: openebs.io/local\n volume.kubernetes.io/selected-node: openebs-worker-1\n Finalizers: [kubernetes.io/pvc-protection]\n Capacity: 5G\n Access Modes: RWO\n VolumeMode: Filesystem\n Mounted By: local-test-0\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal WaitForFirstConsumer 13s persistentvolume-controller waiting for first consumer to be created before binding\n Normal ExternalProvisioning 13s (x2 over 13s) persistentvolume-controller waiting for a volume to be created, either by external provisioner \"openebs.io/local\" or manually created by system administrator\n Normal Provisioning 13s openebs.io/local_openebs-localpv-provisioner-6fc9664557-njvbg_884920b8-e474-4635-9daf-8b4a8f113b10 External provisioner is provisioning volume for claim \"default/openebs-localpv-hostpath-local-test-0\"\n Normal ProvisioningSucceeded 11s openebs.io/local_openebs-localpv-provisioner-6fc9664557-njvbg_884920b8-e474-4635-9daf-8b4a8f113b10 Successfully provisioned volume pvc-b89a1c70-beeb-4bdc-a555-348b01832443\n\nYou can cleanup the whole setup using:\n\n kubectl delete -f openebs-localpv-stateful.yaml\n kubectl delete -f openebs-lite-sc.yaml\n kubectl delete -f openebs-operator-lite-ppc64le.yaml\n\n### About the author:\n\nPeeyush Gupta is a cloud enthusiast with 5+ years of experience in developing cloud platforms and helping customers migrate their legacy applications to the cloud. He has also been a speaker at multiple meetups and loves to contribute to open-source projects. He is currently working with DigitalOcean as Sr. Developer Advocate.\n","notHasFeatureImage":false,"slug":"kubernetes-statefulset-on-ppc64le-using-openebs-localpv-provisioner"},{"id":29,"title":"Data Migration Within Kubernetes Clusters","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"02-07-2020","tags":["Kubernetes"],"excerpt":"In this blog, we'll talk about migrating data within Kubernetes from one node to another without any downtime of the application.","content":"\nIn large scale environments, storage is one of the hard things to manage, and it will be the most crucial component as it has DATA with it. OpenEBS, leading open source Cloud Native Storage, makes managing storage easier in Kubernetes environments. MayaData, the company behind the OpenEBS project, has the vision of achieving data agility by transforming Kubernetes as a data plane. cStor is one of the storage engines of OpenEBS. \nThis blog is for OpenEBS users, specifically **cStor CSI** users looking to **migrate data within Kubernetes** from one node to another without any downtime of the application.\n\n![Data Migration within Kubernetes from Node2 to Node3](/images/blog/Data-migration-diagram.png)\n\n### Create cStor Pools(CSPC):\n\nCreate cStor pools by following the steps mentioned [here](https://github.com/openebs/cstor-operators/blob/master/docs/quick.md). Once the pools are created, wait till all the cStor pools marked as Healthy. Check the cStor pools status by executing `kubectl get cspc -n openebs` command(cspc - cStorPoolCluster)\n\n NAME HEALTHYINSTANCES PROVISIONEDINSTANCES DESIREDINSTANCES AGE\n cspc-stripe-pool 3 3 3 4m13s\n\nFollowing command `kubectl get cspi -n openebs`(cStorPoolInstances) will be helpful to know more information about the pool.\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY READONLY TYPE STATUS AGE\n cspc-stripe-pool-6qkw e2e1-node1 230k 9630M 10G false stripe ONLINE 21m\n cspc-stripe-pool-pn9p e2e1-node2 230k 9630M 10G false stripe ONLINE 12s\n cspc-stripe-pool-psz5 e2e1-node3 230k 9630M 10G false stripe ONLINE 21m\n\n### Create CSI Volume:\n\nCreate CSI volumes on cStor pools created above by following the steps mentioned [here](https://github.com/openebs/cstor-csi#provision-a-cstor-volume-using-openebs-cstor-csi-driver). As part of volume provisioning, a resource called `CStorVolumeConfig` will be created. Once the volume is provisioned successfully, then CVC(cStorVolumeConfig) status will update to Bound, which means all the CStorVolume resources are successfully created. Following is the command which will help to get CVC status `kubectl get cvc -n openebs`.\n\n NAME STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306 Bound 5m56s\n \n\n### **Inspect CVC**:\n\nInterfere CVC to know on which node data exists. When we do `kubectl get cvc -n -o yaml`.\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorVolumeConfig\n name: pvc-d1b26676-5035-4e5b-b564-68869b023306\n …\n …\n spec:\n capacity:\n storage: 5Gi\n ...\n ...\n policy:\n replicaPoolInfo:\n - poolName: cspc-stripe-pool-6qkw\n - poolName: cspc-stripe-pool-pn9p\n status:\n phase: Bound\n poolInfo:\n - cspc-stripe-pool-6qkw\n - cspc-stripe-pool-pn9p\n\nFrom the above info CStorVolumeReplicas(CVR) are scheduled on cStor pools **cspc-stripe-pool-6qkw** and **cspc-stripe-pool-pn9p** from the **status.poolInfo** since above pools are on **e2e1-node1** and **e2e1-node2** (able to find node info from cspi output) sodata also will be available on the same node. Info under spec i.e **spec.policy.replicaPoolInfo** will convey where to schedule cStorVolumeReplicas.\n\nTo know more details of CVR we can get from `kubectl get cvr -n openebs`\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.47G 1.26G Healthy 15h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.47G 1.26G Healthy 15h\n\n**Steps to migrate data from one node to other:**\n\n1. Scale the CStorVolumeReplica to the desired Node.\n2. Scale down the CStorVolumeReplicas from unwanted Node.\n\n**Step1: Scale the CStorVolumeReplicas to the desired node**\n\nGet cStor pool name, which doesn’t have corresponding volume CVR in it and add it under **spec.policy.replicaPoolInfo** of CVC.\n\nWe can get pool name name which doesn’t have CVR in it by comparing outputs of `kubectl get cspi -n openebs -l openebs.io/cstor-pool-cluster=` and `kubectl get cvc -o yaml` as stated inspect CVC. In this example CVR of volume pvc-d1b26676-5035-4e5b-b564-68869b023306 doesn’t not exist in cStor pool **cspc-stripe-pool-psz5.** After finding the pool name add it under **policy.replicaPoolInfo** list in CVC.\n\n apiVersion: cstor.openebs.io/v1\n kind: CStorVolumeConfig\n name: pvc-d1b26676-5035-4e5b-b564-68869b023306\n …\n …\n spec:\n …\n ...\n policy:\n replicaPoolInfo:\n - poolName: cspc-stripe-pool-6qkw\n - poolName: cspc-stripe-pool-pn9p\n - poolName: cspc-stripe-pool-psz5\n …\n …\n status:\n poolInfo:\n - cspc-stripe-pool-6qkw\n - cspc-stripe-pool-pn9p\n - cspc-stripe-pool-psz5\n\n**Superb!**\nOnce the pool was added into the `spec.replicaPoolInfo` then the status of CVC will be updated with a new pool name as shown above, and raise an event which means that CVR was created on a newly added pool. We can get the CVR status by executing `kubectl get cvr -n openebs`\n\n**Events**: Events on corresponding CVC\n\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal ScalingVolumeReplicas 14s (x2 over 15h) cstorvolumeclaim-controller successfully scaled volume replicas to 3\n\n**CVR status(by executing the command)**:\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.48G 1.26G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-psz5 91.4M 42.4M ReconstructingNewReplica 33s\n\nIn the above output, newly created CVRs convey that it reconstructed data from scratch by talking to peer replicas. Wait till the newly created CVR marked as **Healthy**. To know status periodically execute `kubectl get cvr -n openebs` command.\n\n NAME USED ALLOCATED STATUS AGE\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-6qkw 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-pn9p 1.48G 1.25G Healthy 16h\n pvc-d1b26676-5035-4e5b-b564-68869b023306-cspc-stripe-pool-psz5 1.48G 1.25G Healthy 5m28s\n\nNote:\n\n- Reconstructing will take time, depending on the amount of data it has to rebuild.\n\n**Step2: Scale down the CStorVolumeReplicas from unwanted nodes**\n\nOnce the newly created CVR is marked as Healthy, then we can remove the unwanted pool name from Spec of CVC replicaInfo and save it. \n\nIn this example, I need to remove the data from the pool **cspc-stripe-pool-pn9p**, scheduled on **e2e1-node2**. Once the pool name is removed from CVC **spec.policy.replicaPoolInfo**, then corresponding CVR in that pool will be deleted. CVC will generate events and status of CVC also will be updated.\n\nEvents on CVR:\n\n Events:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Warning ScalingVolumeReplicas 4s (x2 over 64m) cstorvolumeclaim-controller Scaling down volume replicas from 3 to 2 is in progress\n Normal ScalingVolumeReplicas 4s (x2 over 64m) cstorvolumeclaim-controller successfully scaled volume replicas to 2\n\nFrom output of `kubectl get cspi -n openebs`\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY READONLY TYPE STATUS AGE\n cspc-stripe-pool-6qkw e2e1-node1 1260M 8370M 9630M false stripe ONLINE 17h\n cspc-stripe-pool-pn9p e2e1-node2 230k 9630M 9630M false stripe ONLINE 16h\n cspc-stripe-pool-psz5 e2e1-node3 1260M 8370M 9630M false stripe ONLINE 17h\n\n**Perfect!**\n\nFrom the above storage usage, I can successfully migrate the data from one node to another without any downtime of the application.\n","notHasFeatureImage":false,"slug":"data-migration-within-kubernetes-clusters"},{"id":30,"title":"Restricting cStor pool usage within a specified threshold value","author":"Giridhara Prasad","author_info":"Lead Engineer at Mayadata Inc. Giridhar is experienced in software test automation, chaos engineering. Currently, he's working on Litmus, an Open Source chaos engineering project.","date":"20-05-2020","tags":["OpenEBS","cStor"],"excerpt":"Learn how to restrict cStor pool usage within a specified threshold value","content":"\ncStor is one of the storage engines provided by OpenEBS. The integral component of the cStor engine is its storage pool from which the volumes are created. The storage pool is constructed with the collection of block devices. When the pool is completely utilized, it may misbehave in such a way that the pool itself cannot be imported successfully to recover from failures.\n\nIn order to overcome this situation, cStor recommends the optimal usage of storage capacity in the pool by restricting the write I/Os by converting it into read-only when the threshold limit is exceeded. While creating cStor SPC, the field *roThresholdLimit* has to be specified in percentage value under pool spec as follows:\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-pool\n type: disk\n poolSpec:\n poolType: striped\n roThresholdLimit: 80\n blockDevices:\n blockDeviceList:\n - blockdevice-31e0768585cb80ed2352affa73ec94e2\n - blockdevice-ab636ddeba8f8cd45f7e91a6b55c15e5\n - blockdevice-75275112e966e43c2ac1311a7a492fac\n\nIn the above snippet, *roThresholdLimit: 80*  indicates that the pool will become read-only when the usage exceeds 80% of its total capacity. Upon trying to create the above SPC, the following CSPs will be created.\n\n NAME ALLOCATED FREE CAPACITY STATUS READONLY TYPE AGE\n cstor-pool-0vl0 45G 4.7G 49.8G Healthy true striped 2m8s\n cstor-pool-qnm1 77K 49.7G 49.8G Healthy false striped 2m8s\n cstor-pool-x4gj 77K 49.7G 49.8G Healthy false striped 2m8s\n\nThe read-only status of each CSP is indicated, as shown in the above template.\n\nWhen the usage of the pool crosses 80% of its actual capacity, it will become read-only. It means all the replicas in that pool won’t serve further write IOs irrespective of the amount of space they consumed. As an impact, if the persistent volumes do not have enough healthy storage replicas, then the volume will become read-only.\n\nThe status of active replica where write IOs were happening turns offline when the pool becomes read-only whereas, for the idle replica, it remains healthy. The sample output of CVR is as below. Here, one replica is in Offline state as its pool is in a read-only state.\n\n k8s@master:kubectl get cvr -n openebs\n NAME USED ALLOCATED STATUS AGE\n pvc-261d6832-8b23-476c-8aa3-b95104e20030-cstor-pool-0vl0 1.15G 1.04G Offline 23m\n pvc-f38f5517-a7bc-492d-a6eb-27ac510ced3b-cstor-pool-qnm1 74.7K 17.7K Healthy 23m56s\n\nThe *roThresholdLimit* is the configurable value. In case, if you want to increase/decrease the percentage value, it has to be modified at each CSP level. Editing at SPC config won’t be effective. In case if the administrator didn’t set the *roThresholdLimit* field during SPC creation, the default value would be set to, 85% considering it as the optimal value for usage. Administrators can set the roThresholdLimit value in CSP from 0 to 100 though the OpenEBS team won't recommend setting it to 100 percent.\nWhen the pool became read-only, the administrator can either increase the pool capacity by executing the steps specified [here](https://github.com/openebs/openebs-docs/blob/day_2_ops/docs/cstor_add_disks_to_spc.md) or increase the roThresholdLimit value in that CSP to make pool RW.\n","notHasFeatureImage":false,"slug":"restricting-cstor-pool-usage-within-a-specified-threshold-value"},{"id":31,"title":"Getting started with K3s in vSphere and OpenEBS cStor","author":"Giridhara Prasad","author_info":"Lead Engineer at Mayadata Inc. Giridhar is experienced in software test automation, chaos engineering. Currently, he's working on Litmus, an Open Source chaos engineering project.","date":"26-03-2020","tags":["OpenEBS"],"excerpt":"In this blog, more of a tutorial, I will walk you through the steps to install K3OS and setup OpenEBS.","content":"\n[K3OS](https://github.com/rancher/k3os/) is a Linux distribution built to run lightweight Kubernetes clusters called [K3s](https://github.com/rancher/k3s/). It is specifically designed only to have what is needed to run [k3s](https://github.com/rancher/k3s).\n\nIn this blog, more of a tutorial, I will walk you through the steps to install K3OS and setup OpenEBS, a CNCF project, and leading Open Source Container Attached Storage solution for Kubernetes Stateful Workloads.\n\n#### **Setting up K3OS in vSphere**\n\nK3OS kernel is forked from Ubuntu-18.04 LTS, and its userspace binaries are from alpine. So, you need to select Ubuntu Linux (64 bit) as the guest operating system while creating a virtual machine.\n\n![Select guest operating system](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage.png)\n\nDownload the latest K3OS iso file (currently v0.9.0) from its [GitHub release](https://github.com/rancher/k3os/releases) page. Attach the iso file into a virtual machine and start it with the live installation option, as shown below.\n\nSelect the option *K3OS LiveCD & install* and boot the operating system.\n\n![Live installation](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-3.png)\n\nAfter booting up successfully, you will be landed in a login prompt. The default user in K3OS is rancher. You can login as rancher user without any password.\n\n![Login prompt](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-6.png)\n\nAfter performing a live install, You need to install the Operating System into a disk and can configure the machine either as a server(Master) or an agent(worker). This can be performed by executing the command `sudo k3os install`.\n\nSelect option 1. Install to disk to install K3OS into the disk. In the preceding questions, set up a new password for rancher user for enabling ssh communication to the server.\n\n![Installing into disk](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-4.png)\n\n### \n**Installing into disk**\n\nYou need to select either server or agent to install the relevant components in the machine. Select 1.server to deploy K3s server components. You can set up a token or cluster secret that could be used while joining K3s agents to the server.\n\n![server installation](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-7.png)\n\nAfter completing the installation, a screen similar to the following one will be displayed.\n\n![Login prompt](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-1.png)\n\nThus, the K3s server can be configured successfully. In case if DHCP is not configured, you need to assign an IP address and other networking details using connmanctl utility. Login into the server as rancher user and enter the password configured in the previous step.\n\nLet us find the connman network service bound to the eth0 device by executing the below command.\n\n sudo connmanctl services\n\nThe above command will list the services below.\n\n![connmanctl services](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-5.png)\n\nAfter identifying the service, you can assign the IP address, netmask, gateway, and DNS server through the following command.\n\n sudo connmanctl config --ipv4 manual --nameservers \n\nAfter executing the above command, ensure if the network is configured correctly through `ifconfig` command.\n\nReboot the machine after setting up networking.\n\n#### **Install K3s agent**\n\nIn K3s nomenclature, Kubernetes workers are called as agents. While installing k3os into a disk, you need to select the option 2. agent to configure K3s agent in the machine.\n\n![Run K3s agent](/images/blog/114---getting-started-with-k3s-in-vsphere-and-use-openebs-cstor-for-its-persistent-storage-2.png)\n\nAfter selecting Agent, you need to provide the URL of the server to which the agent has to be configured. The URL of the k3s server could be formed in the following way.\n\n https://:6443\n\nAfter entering the URL, you need to provide the cluster secret, which was configured during server installation.\n\nAfter providing all the above inputs, initiate the agent deployment.\n\nConfigure networking in the same way as performed above for server and reboot the machine.\n\nAfter rebooting the agent machine, check the cluster status in the server as follows.\n\n k3os-1374 [~]$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n k3os-1374 Ready master 10m v1.17.2+k3s1\n k3os-15360 Ready 10m v1.17.2+k3s1\n k3os-1091 Ready 10m v1.17.2+k3s1\n\nCheck if all the cluster components are configured successfully and all the pods are running successfully by executing the below command.\n\n k3os-1374 [~]$ kubectl get pods -n kube-system\n NAME READY STATUS RESTARTS AGE\n helm-install-traefik-nmjvj 0/1 Completed 0 3d\n svclb-traefik-gp9ff 2/2 Running 1 2d23h\n svclb-traefik-qgdlx 2/2 Running 0 2d23h\n local-path-provisioner-58fb86bdfd-wkdtm 1/1 Running 1 3d\n metrics-server-6d684c7b5-mrxsr 1/1 Running 0 3d\n svclb-traefik-c4v7l 2/2 Running 0 3d\n coredns-d798c9dd-td5tr 1/1 Running 0 3d\n traefik-6787cddb4b-n57jz 1/1 Running 0 3d\n\n#### **Install OpenEBS**\n\nOpenEBS is a CNCF project delivering persistent block storage to the workloads deployed in Kubernetes. [cStor](/docs/concepts/cstor?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807) is one of the storage engines provided by OpenEBS besides [Jiva](/docs/concepts/jiva?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807) and [Local PV](/docs/concepts/localpv?__hstc=216392137.6a5433d986ca5a9bb31cbcea3a03df67.1585216160857.1585216160857.1585216160857.1&__hssc=216392137.1.1585216160858&__hsfp=170476807).\n\ncStor was not supported in K3OS till k3os-v0.8.0 due to this [issue](https://github.com/rancher/k3os/issues/151). This issue has been addressed in v0.9.0 by adding udev support.\n\n k3os-1374 [~]$ kubectl apply -f openebs-operator-1.7.0.yaml \n namespace/openebs created\n serviceaccount/openebs-maya-operator created\n clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created\n clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created\n deployment.apps/maya-apiserver created\n service/maya-apiserver-service created\n deployment.apps/openebs-provisioner created\n deployment.apps/openebs-snapshot-operator created\n configmap/openebs-ndm-config created\n daemonset.apps/openebs-ndm created\n deployment.apps/openebs-ndm-operator created\n deployment.apps/openebs-admission-server created\n deployment.apps/openebs-localpv-provisioner created\n\nCheck if all the OpenEBS components are running successfully.\n\n k3os-1374 [~]$ kubectl get pods -n openebs\n NAME READY STATUS RESTARTS AGE\n openebs-admission-server-f67f77588-8kl78 1/1 Running 0 65s\n openebs-provisioner-7b8c68bf44-7bjw8 1/1 Running 0 66s\n openebs-ndm-qp26v 1/1 Running 0 66s\n openebs-ndm-84zb4 1/1 Running 0 66s\n openebs-ndm-dzghs 1/1 Running 0 66s\n openebs-localpv-provisioner-5c87bbd974-55486 1/1 Running 0 65s\n openebs-ndm-operator-5fccfb7976-dvhj6 1/1 Running 0 66s\n openebs-snapshot-operator-6c4c64d4bc-qxdwd 2/2 Running 0 66s\n maya-apiserver-84785d7fbd-ck7sh 1/1 Running 0 66s\n\nOpenEBS cStor engine requires external disks to be attached to the agents which group to form cStor Pools.\n\nThe disks or block devices are managed by the component called *Node disk manager*, shortly called as *NDM*. After attaching the disks to agent machines, check the block devices by executing the following command.\n\n k3os-1374 [~]$ kubectl get blockdevices -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n blockdevice-30a3eb18f5b9e2d470de45e39f1036b0 k3os-15360 17179869184 Unclaimed Active 1h\n blockdevice-86fc964305abe8970fc1508538a61dbc k3os-1374 17179869184 Unclaimed Active 1h\n blockdevice-b8735721689d5843bca10e7028f60a4e k3os-1091 17179869184 Unclaimed Active 1h\n\nIn this case, one block device has been attached to each K3s agent machine. Let us populate these block devices in the below pool creation manifest under *spec.blockDevices* and create the pool.\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-pool\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 2Gi\n - name: PoolResourceLimits\n value: |-\n memory: 4Gi\n spec:\n name: cstor-disk-pool\n type: disk\n poolSpec:\n poolType: striped\n blockDevices:\n blockDeviceList:\n - blockdevice-30a3eb18f5b9e2d470de45e39f1036b0\n - blockdevice-86fc964305abe8970fc1508538a61dbc \n - blockdevice-b8735721689d5843bca10e7028f60a4e\n\nAfter applying the above definition, check if the pools are created successfully by executing the following command.\n\n k3os-1374 [~]$ kubectl get csp\n NAME ALLOCATED FREE CAPACITY STATUS TYPE AGE\n cstor-disk-pool-rgy3 270K 15.9G 15.9G Healthy striped 2m3s\n cstor-disk-pool-gij4 270K 15.9G 15.9G Healthy striped 2m2s\n cstor-disk-pool-09l1 270K 15.9G 15.9G Healthy striped 2m2s\n\nAll the pool instances should be healthy and each instance runs a pod which can be found by executing the following command.\n\n k3os-1374 [~]$ kubectl get pods -n openebs -l app=cstor-pool\n NAME READY STATUS RESTARTS AGE\n cstor-disk-pool-rgy3-57f965b48c-srz2x 3/3 Running 0 8m33s\n cstor-disk-pool-gij4-77bb4b8f44-s6k89 3/3 Running 0 8m33s\n cstor-disk-pool-09l1-56d444996b-m698h 3/3 Running 0 8m33s\n\nAfter creating cStor pool, we can proceed to create volume. For illustration, let us deploy a busybox with cstor volume as its persistent storage. Before creating a pool, we need to create a storage class specifying the storagePoolClaim that was created in the above step as follows:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-cstor\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk-pool\"\n - name: ReplicaCount\n value: \"3\"\n provisioner: openebs.io/provisioner-iscsi\n\nPopulate storagePoolClaim and ReplicaCount as per your requirement in the above definition and create storage class. You have to use this storage class while creating PVC.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: openebs-pvc\n spec:\n storageClassName: openebs-cstor\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10Gi\n ---\n apiVersion: v1\n kind: Service\n metadata:\n labels:\n name: busybox\n name: busybox\n spec:\n clusterIP: None\n selector:\n app: busybox\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: busybox\n labels:\n app: busybox\n spec:\n selector:\n matchLabels:\n app: busybox\n template:\n metadata:\n labels:\n app: busybox\n spec:\n containers:\n - name: app-busybox\n imagePullPolicy: IfNotPresent\n image: busybox\n command: [\"/bin/sh\"]\n args: [\"-c\", \"while true; do sleep 10;done\"]\n env:\n volumeMounts:\n - name: data-vol\n mountPath: /busybox\n volumes:\n - name: data-vol\n persistentVolumeClaim:\n claimName: openebs-pvc\n\nAfter updating the storage class in the above manifest, let us deploy the busybox application by applying the above definition.\n\nCheck if the PVC is created and mounted successfully on the application pod by checking their status.\n\n k3os-1374 [~]$ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n openebs-pvc Bound pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d 10Gi RWO openebs-cstor 4m35s\n \n k3os-1374 [~]$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n busybox-748fb77c75-9lwzz 1/1 Running 0 4m42s\n\nAs we specified ReplicaCount as 3 in the storage class, 3 volume replicas will be created which can be found by executing the following command.\n\n k3os-1374 [~]$ kubectl get cvr -n openebs\n NAME USED ALLOCATED STATUS AGE\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-09l1 7.95M 116K Healthy 6m37s\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-rgy3 7.95M 116K Healthy 6m37s\n pvc-6cd2b30a-49ed-4605-b1e0-dd23c45e548d-cstor-disk-pool-gij4 7.95M 116K Healthy 6m37s\n\nThus, the cStor engine can be used to provision persistent volume for the workloads in K3s.\n\nPlease leave your valuable comments or feedback in the comment section below if you find this tutorial helpful.\n","notHasFeatureImage":false,"slug":"getting-started-with-k3s-in-vsphere-and-openebs-cstor"},{"id":32,"title":"Resizing the ZFS-LocalPV Volumes","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"26-03-2020","tags":["LocalPV","OpenEBS"],"excerpt":"In this post, we will focus on how we can resize the volumes provisioned by ZFS-LocalPV without restarting the application.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can resize the volumes provisioned by ZFS-LocalPV without restarting the application.\n\n### **Prerequisite**\n\nPlease make sure you have installed the ZFS-LocalPV Driver version v0.5 or later:\n\n $ kubectl apply -f\n https://raw.githubusercontent.com/openebs/zfs-localpv/v0.5.x/deploy/zfs-operator.yaml\n\nMake sure you are using k8s version 1.16+ as this feature is in beta. In Kubernetes 1.14 and 1.15, this feature was in alpha status and required enabling the following feature gate:\n\n --feature-gates=ExpandCSIVolumes=true\n\nAlso for Kubernetes 1.14 and 1.15, online expansion feature gate has to be enabled explicitly:\n\n --feature-gates=ExpandInUsePersistentVolumes=true\n\n### **Introduction**\n\nThe ZFS-LocalPV CSI driver supports ONLINE volume expansion, which means, if the application is using the volume, you can perform the volume expansion. This also means that the volume expansion will be performed when an application is using that volume. So, if an application is not running and we have performed the resize operation, the Driver will wait for the application to start for the resize operation to complete.\n\n### **Setup**\n\nCreate the StorageClass with allowVolumeExpansion as true. We can resize only those volumes which are using the StorageClass with this flag as true.\n\n $ cat sc.yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-zfspv\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n $ kubectl apply -f sc.yaml\n storageclass.storage.k8s.io/openebs-zfspv created\n\nCreate the PVC using the above StorageClass:\n\n $ cat pvc.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: csi-zfspv\n spec:\n storageClassName: openebs-zfspv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n $ kubectl apply -f pvc.yaml\n persistentvolumeclaim/csi-zfspv created\n\nNow deploy the application using the above PVC. Here, we will be using below Percona application:\n\n $ cat percona.yaml\n apiVersion: v1\n kind: ConfigMap\n metadata:\n annotations:\n name: sqltest\n namespace: default\n data:\n sql-test.sh: |\n #!/bin/bash\n DB_PREFIX=\"Inventory\"\n DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`\n DB_NAME=\"${DB_PREFIX}_${DB_SUFFIX}\"\n echo -e \"\\nWaiting for mysql server to start accepting connections..\"\n retries=10;wait_retry=30\n for i in `seq 1 $retries`; do\n mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1\n rc=$?\n [ $rc -eq 0 ] && break\n sleep $wait_retry\n done\n if [ $rc -ne 0 ];\n then\n echo -e \"\\nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exiting\\n\"\n exit 1\n fi\n mysql -uroot -pk8sDem0 -e \"CREATE DATABASE $DB_NAME;\"\n mysql -uroot -pk8sDem0 -e \"CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"INSERT INTO Hardware (id, name, owner, description) values (1, \"dellserver\", \"basavaraj\", \"controller\");\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"DROP DATABASE $DB_NAME;\"\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: percona\n labels:\n name: percona\n spec:\n replicas: 1\n selector:\n matchLabels:\n name: percona\n template:\n metadata:\n labels:\n name: percona\n spec:\n containers:\n - resources:\n name: percona\n image: openebs/tests-custom-percona:latest\n imagePullPolicy: IfNotPresent\n args:\n - \"--ignore-db-dir\"\n - \"lost+found\"\n env:\n - name: MYSQL_ROOT_PASSWORD\n value: k8sDem0\n ports:\n - containerPort: 3306\n name: percona\n volumeMounts:\n - mountPath: /var/lib/mysql\n name: demo-vol1\n - mountPath: /sql-test.sh\n subPath: sql-test.sh\n name: sqltest-configmap\n livenessProbe:\n exec:\n command: [\"bash\", \"sql-test.sh\"]\n initialDelaySeconds: 30\n periodSeconds: 1\n timeoutSeconds: 10\n volumes:\n - name: demo-vol1\n persistentVolumeClaim:\n claimName: csi-zfspv\n - name: sqltest-configmap\n configMap:\n name: sqltest\n ---\n apiVersion: v1\n kind: Service\n metadata:\n name: percona-mysql\n labels:\n name: percona-mysql\n spec:\n ports:\n - port: 3306\n targetPort: 3306\n selector:\n name: percona\n\nApply the above YAML to deploy the Percona application:\n\n $ kubectl apply -f percona.yaml\n configmap/sqltest created\n deployment.apps/percona created\n service/percona-mysql created\n\nNow the setup is ready and the application is running:\n\n $ kubectl get po\n NAME READY STATUS RESTARTS AGE\n percona-9449b4b9c-48qpw 1/1 Running 0 38s\n\nCheck the volume size at the application size:\n\n $ kubectl exec -it percona-9449b4b9c-48qpw bash\n root@percona-9449b4b9c-48qpw:/# df -h\n Filesystem Size Used Avail Use% Mounted on\n none 91G 18G 69G 21% /\n tmpfs 3.9G 0 3.9G 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sda1 91G 18G 69G 21% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n /dev/zd0 4.9G 234M 4.7G 5% /var/lib/mysql\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nFrom above o/p we can see that the volume has been created of size 5Gi and it is attached to the application at the given mount point (/var/lib/mysql).\n\n### **Volume Resize**\n\nHere, we just have to update the PVC with the new size and apply it. Please note that volume shrinking is not supported, so you have to change the size to a higher value. Here, in our case, we will update the size to 8Gi\n\n $ cat pvc.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: csi-zfspv\n spec:\n storageClassName: openebs-zfspv\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 8Gi\n\nApply the above YAML to perform the resize:\n\n $ kubectl apply -f pvc.yaml\n persistentvolumeclaim/csi-zfspv configured\n\nNow, we can keep checking the PVC for the new size to be updated, it may take a while. Once resize operation is done we can see the PVC output with the updated size:\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-9b5c22a5-29be-428e-aa96-5e183c1c4c62 8Gi RWO openebs-zfspv 33m\n\nWe can also exec into the application pod and verify that the new size is visible to the application:\n\n $ kubectl exec -it percona-9449b4b9c-48qpw bash\n root@percona-9449b4b9c-48qpw:/# df -h\n Filesystem Size Used Avail Use% Mounted on\n none 91G 18G 69G 21% /\n tmpfs 3.9G 0 3.9G 0% /dev\n tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup\n /dev/sda1 91G 18G 69G 21% /etc/hosts\n shm 64M 0 64M 0% /dev/shm\n /dev/zd0 7.9G 237M 7.6G 3% /var/lib/mysql\n tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount\n tmpfs 3.9G 0 3.9G 0% /sys/firmware\n\nI hope you find this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"resizing-the-zfs-localpv-volumes"},{"id":33,"title":"Snapshot and Clone for ZFS LocalPV","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"03-03-2020","tags":["OpenEBS","ZFS","Open Source"],"excerpt":"In this post, we will focus on how we can create a snapshot and clone for volumes provisioned by ZFS-LocalPV.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.2b738ae93497639f7465a332e1aef247.1584602510099.1584602510099.1584602510099.1&__hssc=216392137.1.1584602510100&__hsfp=2870217423) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can create a snapshot and clone for volumes provisioned by ZFS-LocalPV.\n\n#### **Prerequisite**\n\nFor clone, we need to have VolumeSnapshotDataSource support, which is in beta in Kubernetes 1.17. If you are using the Kubernetes version less than 1.17, you have to enable the VolumeSnapshotDataSource feature gate at kubelet and kube-apiserver.\n\n#### **Snapshot**\n\nWe can create a snapshot of a volume that can be used further for creating a clone and for taking a backup. To create a snapshot, we have to first create a SnapshotClass just like a storage class where you can provide deletionPolicy as Retain or Delete.\n\n $ cat snapshotclass.yaml\n kind: VolumeSnapshotClass\n apiVersion: snapshot.storage.k8s.io/v1beta1\n metadata:\n name: zfspv-snapclass\n annotations:\n snapshot.storage.kubernetes.io/is-default-class: \"true\"\n driver: zfs.csi.openebs.io\n deletionPolicy: Delete\n\nApply the snapshotclass YAML:\n\n $ kubectl apply -f snapshotclass.yaml\n volumesnapshotclass.snapshot.storage.k8s.io/zfspv-snapclass created\n\nFind a PVC for which snapshot has to be created\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 4Gi RWO openebs-zfspv 2m35s\n\nCreate the snapshot using the created SnapshotClass for the selected PVC\n\n $ cat snapshot.yaml\n apiVersion: snapshot.storage.k8s.io/v1beta1\n kind: VolumeSnapshot\n metadata:\n name: zfspv-snap\n spec:\n volumeSnapshotClassName: zfspv-snapclass\n source:\n persistentVolumeClaimName: csi-zfspv\n\nApply the snapshot.yaml\n\n $ kubectl apply -f snapshot.yaml\n volumesnapshot.snapshot.storage.k8s.io/zfspv-snap created\n\nPlease note that you have to create the snapshot in the same namespace where the PVC is created. Check the created snapshot resource, make sure readyToUsefield is true, before using this snapshot for any purpose.\n\n $ kubectl get volumesnapshot.snapshot\n NAME AGE\n zfspv-snap 2m8s\n \n $ kubectl get volumesnapshot.snapshot zfspv-snap -o yaml\n apiVersion: snapshot.storage.k8s.io/v1beta1\n kind: VolumeSnapshot\n metadata:\n annotations:\n kubectl.kubernetes.io/last-applied-configuration: |\n {\"apiVersion\":\"snapshot.storage.k8s.io/v1beta1\",\"kind\":\"VolumeSnapshot\",\"metadata\":{\"annotations\":{},\"name\":\"zfspv-snap\",\"namespace\":\"default\"},\"spec\":{\"source\":{\"persistentVolumeClaimName\":\"csi-zfspv\"},\"volumeSnapshotClassName\":\"zfspv-snapclass\"}}\n creationTimestamp: \"2020-02-25T08:25:51Z\"\n finalizers:\n - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection\n - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\n generation: 1\n name: zfspv-snap\n namespace: default\n resourceVersion: \"447494\"\n selfLink: /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/zfspv-snap\n uid: 3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n spec:\n source:\n persistentVolumeClaimName: csi-zfspv\n volumeSnapshotClassName: zfspv-snapclass\n status:\n boundVolumeSnapshotContentName: snapcontent-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n creationTime: \"2020-02-25T08:25:51Z\"\n readyToUse: true\n restoreSize: \"0\"\n\nCheck the OpenEBS resource for the created snapshot. Check, status should be Ready.\n\n $ kubectl get zfssnap -n openebs\n NAME AGE\n snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 3m32s\n \n $ kubectl get zfssnap snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd -n openebs -oyaml\n apiVersion: openebs.io/v1alpha1\n kind: ZFSSnapshot\n metadata:\n creationTimestamp: \"2020-02-25T08:25:51Z\"\n finalizers:\n - zfs.openebs.io/finalizer\n generation: 2\n labels:\n kubernetes.io/nodename: e2e1-node2\n openebs.io/persistent-volume: pvc-73402f6e-d054-4ec2-95a4-eb8452724afb\n name: snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n namespace: openebs\n resourceVersion: \"447328\"\n selfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/zfssnapshots/snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n uid: 6142492c-3785-498f-aa4a-569ec6c0e2b8\n spec:\n capacity: \"4294967296\"\n fsType: zfs\n ownerNodeID: e2e1-node2\n poolName: test-pool\n volumeType: DATASET\n status:\n state: Ready\n\nWe can go to the node and confirm that snapshot has been created:\n\n # zfs list -t all\n NAME USED AVAIL REFER MOUNTPOINT\n test-pool 818K 9.63G 24K /test-pool\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 24K 4.00G 24K /var/lib/kubelet/pods/3862895a-8a67-446e-80f7-f3c18881e391/volumes/kubernetes.io~csi/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb/mount\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 0B - 24K -\n\n#### **Clone**\n\nWe can create a clone volume from a snapshot and use that volume for some application. We can create a PVC YAML and mention the snapshot name in the datasource.\n\n $ cat clone.yaml\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: zfspv-clone\n spec:\n storageClassName: openebs-zfspv\n dataSource:\n name: zfspv-snap\n kind: VolumeSnapshot\n apiGroup: snapshot.storage.k8s.io\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nThe above YAML says that create a volume from the snapshot zfspv-snap. Applying the above YAML will create a clone volume on the same node where the original volume is present. The newly created clone PV will also be there on the same node where the original PV is there. Apply the clone YAML\n\n $ kubectl apply -f clone.yaml \n persistentvolumeclaim/zfspv-clone created\n\nNote that the clone PVC should also be of the same size as that of the original volume. Currently resize is not supported. Also, note that the poolname should also be same, as across the ZPOOL clone is not supported. So, if you are using a separate storageclass for the clone PVC, please make sure it refers to the same ZPOOL.\n\n $ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n csi-zfspv Bound pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 4Gi RWO openebs-zfspv 13m\n zfspv-clone Bound pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 4Gi RWO openebs-zfspv 34s\n\nWe can see in the above output that zfspv-clone claim has been created and it is bound. Also, we can check the zfs list on node and verify that clone volume is created.\n\n $ zfs list -t all\n NAME USED AVAIL REFER MOUNTPOINT\n test-pool 834K 9.63G 24K /test-pool\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb 24K 4.00G 24K /var/lib/kubelet/pods/3862895a-8a67-446e-80f7-f3c18881e391/volumes/kubernetes.io~csi/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb/mount\n test-pool/pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd 0B - 24K -\n test-pool/pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 0B 9.63G 24K none\n\nThe clone volume will have properties same as snapshot properties which are the properties when that snapshot has been created. The ZFSVolume object for the clone volume will be something like below:\n\n $ kubectl describe zv pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19 -n openebs\n Name: pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19\n Namespace: openebs\n Labels: kubernetes.io/nodename=e2e1-node2\n Annotations: none\n API Version: openebs.io/v1alpha1\n Kind: ZFSVolume\n Metadata:\n Creation Timestamp: 2020-02-25T08:34:25Z\n Finalizers:\n zfs.openebs.io/finalizer\n Generation: 1\n Resource Version: 448930\n Self Link: /apis/openebs.io/v1alpha1/namespaces/openebs/zfsvolumes/pvc-c095aa52-8d09-4bbe-ac3c-bb88a0e7be19\n UID: e38a9f9a-fb76-466b-a6f9-8d070e0bec6f\n Spec:\n Capacity: 4294967296\n Fs Type: zfs\n Owner Node ID: e2e1-node2\n Pool Name: test-pool\n Snapname: pvc-73402f6e-d054-4ec2-95a4-eb8452724afb@snapshot-3cbd5e59-4c6f-4bd6-95ba-7f72c9f12fcd\n Volume Type: DATASET\n Events: none\n\nHere you can note that this resource has Snapname field which tells that this volume is created from that snapshot.\n\nI hope you found this post useful. Feel free to contact me with any feedback or questions by using the comment section below.\n","notHasFeatureImage":false,"slug":"snapshot-and-clone-for-zfs-localpv"},{"id":34,"title":"Setup Continuous Integration for Helm chart","author":"Intakhab Ali","author_info":"Software Engineer at MayaData","date":"05-02-2020","tags":["Helm","OpenEBS","Tutorials","Kubernetes"],"excerpt":"In this blog, we'll set up a continuous integration of the Helm chart. We'll package the Helm chart with the help of CI tools & push them to chart registry.","content":"\n[Helm](https://www.helm.sh/) is a package manager for Kubernetes that allows developers and operators to easily package, configure, and deploy applications and services onto Kubernetes clusters.\n\nHelm is now an official Kubernetes project and is part of the[ Cloud Native Computing Foundation](https://www.cncf.io/), a non-profit Linux Foundation that supports Open Source projects in and around the Kubernetes ecosystem.\n\nIn this tutorial, we will set up a continuous integration of the Helm chart. We will package the Helm chart with the help of CI tools like (Travis, Jenkins), and push it to chart registries like (Harbor, Chartmuseum).\n\n## Prerequisites:\n\n- Registry to store Helm like Harbor or Chartmuseum\n- Understanding of Helm and any of the CI platforms (Travis, Jenkins, circle, CI)\n- A Git repository to maintain version control of helm chart\n\n**I am going to use Travis as a CI platform and Harbor as a Helm registry to host the helm.**\n\n**As I choose Travis here, .travis.yml consists of the job lifecycle. Let’s write job cycle for the helm chart.**\n\n### Lifecycle 1:\n\nChoose the base language as Python\n\n ---\n language: python\n\nWe need to have some environment variables so that we can update whenever there’s a new version of the Helm release or change of the registry URL.\n\nHere is the list of a variable that we’ll need:\n\n***HELM_URL=[https://storage.googleapis.com/kubernetes-helm](https://storage.googleapis.com/kubernetes-helm) (this is the URL where we can download the helm package)***\n\n***HELM_TGZ=helm-v2.4.2-linux-amd64.tar.gz (this is the Helm tar filename)***\n\n***REPO_DIR=/home/travis/build/inyee786/test-helm (this is the path where Travis keep Git folder)***\n\n***YAMLLINT_VERSION=1.8.1 (this is yamllint version which is used to check lint of file)***\n\n***HARBOR_CHART_URL=https://harbor-test.mayadata.io/chartrepo (change this according to your chart registry url{harbor or Chartmuseum}***\n\n***HARBOR_PROJECT_NAME=maya (this is the Harbor project name, where we will store the chart)***\n\n***CHART_FOLDER=charts (this is the folder name, where we can keep the Helm charts)***\n\nIt looks like this\n\n env:\n global:\n - HELM_URL=https://storage.googleapis.com/kubernetes-helm\n - HELM_TGZ=helm-v2.4.2-linux-amd64.tar.gz\n - REPO_DIR=/home/travis/build/inyee786/test-helm\n - YAMLLINT_VERSION=1.8.1\n - HARBOR_CHART_URL=https://harbor-\n test.mayadata.io/chartrepo\n - HARBOR_PROJECT_NAME=maya\n - CHART_FOLDER=charts\n\nWe need some private variables, where we can store the credentials and push it to the Helm registry (Harbor has an excellent feature where we can have bot user, and you can use the bot credential). All we have to feed is-\n\n HARBOR_USERNAME:\n HARBOR_PASSWORD:\n\nInside Travis, go to (****settings > Environment Variables****) to set the private env\n\n![Environment variables](/images/blog/environment-variables.png)\n\n### Lifecycle 2 :\n\nInstall the Prerequisites to Set up a CI environment to build and check the YAML lint.\n\nDownload helm and ****untar**** the chart after downloading\n\n- wget ${HELM_URL}/${HELM_TGZ}\n- tar xzfv ${HELM_TGZ}\n- PATH=`pwd`/linux-amd64/:$PATH\n\nInitialize the helm client and update the helm repo\n\n- Helm init — client-only\n- Helm repo update\n\nInstall helm plugin to push chart on the registry\n\n- Helm plugin install[ https://github.com/chartmuseum/helm-push](https://github.com/chartmuseum/helm-push) — version v0.7.1\n\nInstall yamllint python package to check the lint\n\n- sudo pip install yamllint==”${YAMLLINT_VERSION}”\n\nIt looks like the below config\n\n install:\n # Installing Helm\n - wget ${HELM_URL}/${HELM_TGZ}\n - tar xzfv ${HELM_TGZ}\n - PATH=`pwd`/linux-amd64/:$PATH\n - helm init --client-only\n # helm plugin to push helm chart\n - helm plugin install https://github.com/chartmuseum/helm-\n push --version v0.7.1\n # Installing pip deps\n - sudo pip install yamllint==\"${YAMLLINT_VERSION}\"\n - helm repo update\n\n### Lifecycle 3 :\n\nBefore going further to build a chart, we need to run some script to check the lint in the chart and Travis file. It is a good practice to check the lint\n\nCheck the Helm lint of all Helm chart\n\n- For dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\nhelm lint ${REPO_DIR}/${CHART_FOLDER}/$dir\nif [ $? != 0 ]; then\ntravis_terminate 1\nfi\n\nTo check the YAML lint for travis.yml, chart.yaml and value.yaml, we use the yamllint python package. We need the rule to check the lint.\n\n- yamllint -c .yamllint.yml -s .travis.yml .yamllint.yml\n- yamllint -c .yamllint.yml -s $(find . -type f -name “Chart.yaml”)\n- yamllint -c .yamllint.yml -s $(find . -type f -name “values.yaml”)\n\nThe script section should look like the below config\n\n script:\n # Check charts format\n - >\n for dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\n helm lint ${REPO_DIR}/${CHART_FOLDER}/$dir\n if [ $? != 0 ]; then\n travis_terminate 1\n fi\n done\n # Check YAML styling\n - yamllint -c .yamllint.yml -s .travis.yml .yamllint.yml\n - yamllint -c .yamllint.yml -s $(find . -type f -name \"Chart.yaml\")\n - yamllint -c .yamllint.yml -s $(find . -type f -name \"values.yaml\")\n\nHere comes the interesting part where we are going to build and package the chart.\n\n****Lifecycle 4:****\n\nIt’s better to build and push when we merge the chart in the **master** branch. So we run the below command when we merge the chart in the **master** branch\n\nWe need a temporary directory where we will build and package the chart\n\n- BUILD_DIR=$(mktemp -d)\n\nRun a loop to all the charts to build, package, and push it to the registry. The below commands will run on each chart\n\n- helm dep update ${REPO_DIR}/${CHART_FOLDER}/$dir\n\nPackage the chart with the below command\n\n- helm package ${REPO_DIR}/${CHART_FOLDER}/$dir\n\nThen push the chart to registry\n\n- helm push — username ${HARBOR_USERNAME} — password ${HARBOR_PASSWORD} ${REPO_DIR}/${CHART_FOLDER}/$dir ${HARBOR_CHART_URL}/maya\n\nBelow is the what the config will look like\n\n # Temporary dir for storing new packaged charts and index files\n BUILD_DIR=$(mktemp -d) # Push temporary directory to the stack\n pushd $BUILD_DIR # Iterate over all charts are package them push it to Harbor\n for dir in `ls ${REPO_DIR}/${CHART_FOLDER}`; do\n helm dep update ${REPO_DIR}/${CHART_FOLDER}/$dir\n helm package ${REPO_DIR}/${CHART_FOLDER}/$dir\n helm push --username ${HARBOR_USERNAME} --password ${HARBOR_PASSWORD} ${REPO_DIR}/${CHART_FOLDER}/$dir ${HARBOR_CHART_URL}/maya\n if [ $? != 0 ]; then\n travis_terminate 1\n fi\n done# Pop temporary directory from the stack\n popd\n\nWow! We have successfully completed all the steps. Now, our setup is ready to build and push the helm chart to the registry.\n\n![Project Dashboard](/images/blog/project-dashboard.png)\n\nHere is the full Travis file\n[https://gist.github.com/inyee786/d779f347d7fa272aed4ee8457182af35.js](https://gist.github.com/inyee786/d779f347d7fa272aed4ee8457182af35.js)\n\nHere is .yamllint.yml file which contains lint rule for charts.yaml values.yaml and .travis.yaml\n[https://gist.github.com/inyee786/ef15b05c98bb4761b41af5f4fe268239.js](https://gist.github.com/inyee786/ef15b05c98bb4761b41af5f4fe268239.js)\n\n## Conclusion:\n\nHere we packaged the helm chart and pushed it to the helm registry.\n\n## About me\n\nYou can follow me at the below profiles and can ask any questions related to Angular, JavaScript, Travis, Kubernetes, etc.\n\n- [GitHub](https://github.com/inyee786/)\n- [Linkedin](https://www.linkedin.com/in/intakhab-ali/)\n- [Medium](https://medium.com/@intakhab.cusat)\n\nThis blog was originally published on 28th Jan 2020 on [MayaData’s blog](https://blog.mayadata.io/openebs/setup-continuous-integration-for-helm-chart).\n","notHasFeatureImage":false,"slug":"setup-continuous-integration-for-helm-chart"},{"id":35,"title":"Monitoring ZFS-LocalPV Volumes","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"28-01-2020","tags":["OpenEBS","LocalPV","ZFS"],"excerpt":"In this post, we will focus on how we can set up the Prometheus alert for Provisioned volumes when space utilization has reached a critical point.","content":"\nBefore reading this post, please read my previous [post](/blog/openebs-dynamic-volume-provisioning-on-zfs?__hstc=216392137.7dc0753f698b104ea002a16b84268b54.1580207831486.1580207831486.1580207831486.1&__hssc=216392137.1.1580207831487&__hsfp=818904025) for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can set up the Prometheus alert for Provisioned volumes when space utilization has reached a critical point.\n\n### Prerequisite\n\nMake sure you are using k8s version 1.15+ to access the CSI volume metrics.\n\n### Setup helm\n\nThis step uses helm as the Kubernetes package manager. If you have not setup the helm, execute the below configuration. Otherwise, you can move on to the next step.\n\n $ helm version\n Client: &version.Version{SemVer:\"v2.16.1\", GitCommit:\"bbdfe5e7803a12bbdf97e94cd847859890cf4050\", GitTreeState:\"clean\"}\n Server: &version.Version{SemVer:\"v2.16.1\", GitCommit:\"bbdfe5e7803a12bbdf97e94cd847859890cf4050\", GitTreeState:\"clean\"}\n \n $ helm init\n Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n \n Please note: By default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n To prevent this, run `helm init` with the --tiller-tls-verify flag.\n For more information on securing your installation see: (https://docs.helm.sh/using_helm/#securing-your-helm-installation)[https://docs.helm.sh/using_helm/#securing-your-helm-installation]\n \n $ kubectl create serviceaccount --namespace kube-system tiller\n serviceaccount/tiller created\n \n $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller\n clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created\n \n $ kubectl patch deploy --namespace kube-system tiller-deploy -p '{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}'\n deployment.extensions/tiller-deploy patched\n\n### Install Prometheus Operator\n\nOnce the helm is ready and the related tiller pods are up and running, use the Prometheus chart from the helm repository.\n\n $ helm install stable/prometheus-operator --name prometheus-operator\n\nCheck all the required pods are up and running\n\n $ kubectl get pods -l \"release=prometheus-operator\"\n NAME READY STATUS RESTARTS AGE\n prometheus-operator-grafana-85bb5d49d-bffdg 2/2 Running 0 2m21s\n prometheus-operator-operator-64844759f7-rpwws 2/2 Running 0 2m21s\n prometheus-operator-prometheus-node-exporter-p9rl8 1/1 Running 0 2m21s\n\n### Set up the alert rule\n\nCheck all the rules available in the system:\n\n $ kubectl get PrometheusRule\n NAME AGE\n prometheus-operator-alertmanager.rules 4m21s\n prometheus-operator-etcd 4m21s\n prometheus-operator-general.rules 4m21s\n prometheus-operator-k8s.rules 4m21s\n prometheus-operator-kube-apiserver-error 4m21s\n prometheus-operator-kube-apiserver.rules 4m21s\n prometheus-operator-kube-prometheus-node-recording.rules 4m21s\n prometheus-operator-kube-scheduler.rules 4m21s\n prometheus-operator-kubernetes-absent 4m21s\n prometheus-operator-kubernetes-apps 4m21s\n prometheus-operator-kubernetes-resources 4m21s\n prometheus-operator-kubernetes-storage 4m21s\n prometheus-operator-kubernetes-system 4m21s\n prometheus-operator-kubernetes-system-apiserver 4m21s\n prometheus-operator-kubernetes-system-controller-manager 4m21s\n prometheus-operator-kubernetes-system-kubelet 4m21s\n prometheus-operator-kubernetes-system-scheduler 4m21s\n prometheus-operator-node-exporter 4m21s\n prometheus-operator-node-exporter.rules 4m21s\n prometheus-operator-node-network 4m21s\n prometheus-operator-node-time 4m21s\n prometheus-operator-node.rules 4m21s\n prometheus-operator-prometheus 4m21s\n prometheus-operator-prometheus-operator 4m21s\n\nYou can edit any of the default rules or create a new rule to get the alerts. Below is a sample rule to start generating alerts when available storage space is less than 10%.\n\n apiVersion: monitoring.coreos.com/v1\n kind: PrometheusRule\n metadata:\n labels:\n app: prometheus-operator\n chart: prometheus-operator-8.5.4\n heritage: Tiller\n release: prometheus-operator\n name: prometheus-operator-zfs-alertmanager.rules\n namespace: default\n spec:\n groups:\n - name: zfsalertmanager.rules\n rules:\n - alert: ZFSVolumeUsageCritical\n annotations:\n message: The PersistentVolume claimed by {{ $labels.persistentvolumeclaim\n }} in Namespace {{ $labels.namespace }} is only {{ printf \"%0.2f\" $value\n }}% free.\n expr: |\n 100 * kubelet_volume_stats_available_bytes{job=\"kubelet\"}\n /\n kubelet_volume_stats_capacity_bytes{job=\"kubelet\"}\n < 10\n for: 1m\n labels:\n severity: critical\n\nNow apply the above YAML so that Prometheus can fire the alerts when available space is less than 10%.\n\n### Check the Prometheus alert\n\nTo view the Prometheus web UI, you must expose it through a Service. A simple way to accomplish this is to use a Service of type NodePort.\n\n $ cat prometheus-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n name: prometheus-service\n spec:\n type: NodePort\n ports:\n - name: web\n nodePort: 30090\n port: 9090\n protocol: TCP\n targetPort: web\n selector:\n prometheus: prometheus-operator-prometheus\n\nApply the above YAML\n\n $ kubectl apply -f prometheus-service.yaml\n service/prometheus-service created\n\nNow you can access the alert manager UI via “node’s-external-ip:30090”.\n\n $ kubectl get nodes -owide\n NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\n gke-zfspv-pawan-default-pool-3e407350-xvzp Ready 103m v1.15.4-gke.22 10.168.0.45 34.94.3.140 Ubuntu 18.04.3 LTS 5.0.0-1022-gke docker://19.3.2\n\nHere, we can access the alert manager via URL: [http://34.94.3.140:30090/](http://34.94.3.140:30090/)\n\n### Check the Alert Manager\n\nTo view the Alert Manager web UI, expose it through a Service of type NodePort.\n\n $ cat alertmanager-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n name: alertmanager-service\n spec:\n type: NodePort\n ports:\n - name: web\n nodePort: 30093\n port: 9093\n protocol: TCP\n targetPort: web\n selector:\n alertmanager: prometheus-operator-alertmanager\n\nApply the above YAML\n\n $ kubectl apply -f alertmanager-service.yaml\n service/alertmanager-service created\n\nNow you can access the alert manager UI via “node’s-external-ip:30093”.\n\n $ kubectl get nodes -owide\n NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME\n gke-zfspv-pawan-default-pool-3e407350-xvzp Ready 103m v1.15.4-gke.22 10.168.0.45 34.94.3.140 Ubuntu 18.04.3 LTS 5.0.0-1022-gke docker://19.3.2\n\nAgain, we can access the alert manager via URL: [http://34.94.3.140:30093/.](http://34.94.3.140:30093/)\n\nI hope you found this post to be useful. Feel free to contact me with any feedback or questions by using the comment section below.\n\nThis blog was originally published on [Jan 22, 2020, on the MayaData blog](https://blog.mayadata.io/openebs/monitoring-zfs-localpv-volumes).\n","notHasFeatureImage":false,"slug":"monitoring-zfs-localpv-volumes"},{"id":36,"title":"Uniquely identifying disks in OpenEBS on VMWare platform","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"22-01-2020","tags":["OpenEBS","Vmware","Vcenter","Virtual Disk","Kubernetes"],"excerpt":"A little bit of background. I work at a company called MayaData who develops a very cool Open Source software called OpenEBS (CNCF Sandbox project) that simplifies the deployment of stateful applications on Kubernetes.","content":"\nA little bit of background. I work at a company called [MayaData](https://mayadata.io/) who develops a very cool Open Source software called OpenEBS (CNCF Sandbox project) that simplifies the deployment of stateful applications on Kubernetes. You should check it out at [www.openebs.io](http://www.openebs.io/?__hstc=216392137.84d52389458ef57b0491fddb252202d6.1570688281471.1578466343199.1578469779597.19&__hssc=216392137.2.1578469779597&__hsfp=2854279793).\n\nKubernetes can be installed on any type of machine; be it a Virtual Machine, bare metal, or cloud machine. Kubernetes abstracts away most of the significant bits of a system, except storage. When it comes to storage, the main reason an abstraction will not work is that there is no uniqueness among the storage devices themselves. Every vendor and every virtualization platform implements it differently.\n\nWe hit this issue of unique virtual disks while deploying OpenEBS on Kubernetes backed by VMware VMS or other virtualization platforms because OpenEBS NDM is not able to uniquely identify the block devices themselves.\n\nChanging the absolute configuration on the Virtual Machine can help you get around this issue.\n\nHere are the steps to enable unique disk IDs in VMware via vSphere client:\n\n1. Right-click the virtual machine for which you want to enable the disk UUID attribute, and select Power > Power Off.\n2. The virtual machine powers off.\n3. Right-click the virtual machine, and click Edit Settings.\n4. Click the Options tab, and select the General entry in the settings column.\n5. Click Configuration Parameters. The Configuration Parameters window appears.\n6. Click Add Row.\n7. In the Name column, enter disk.Enable UUID\n8. In the Value column, enter TRUE.\n9. Click OK and click Save.\n10. Power on the virtual machine.\n\nThis will assign WWN to each disk in the Virtual Machine\n\nThat is it for today’s tutorial. If you have any questions, feedback, or any topic that you feel I should cover next, feel free to comment on our blog or reach out to us on our [Slack](https://slack.openebs.io./) channel.\n\nThis blog was originally published on [Oct 01, 2019, on the MayaData blog](https://blog.mayadata.io/openebs/uniquely-identifying-disks-in-openebs-on-vmware-platform).\n","notHasFeatureImage":false,"slug":"uniquely-identifying-disks-in-openebs-on-vmware-platform"},{"id":37,"title":"Creating manual BlockDevice","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"16-01-2020","tags":["Block Devices","Docker","Uncategorized","Disk","OpenEBS","Kubernetes"],"excerpt":"BlockDevices are the consumable units of storage in the OpenEBS storage solution. Users can create BlockDevices manually to support custom partitions/lvms, etc., which are not detected by NDM. To create a manual BlockDevice, follow the steps below","content":"\nBlockDevices are the consumable units of storage in the OpenEBS storage solution. Currently, NDM supports the discovery and management of only a complete disk.\n\nHowever, users can create BlockDevices manually to support custom partitions/lvms, etc., which are not detected by NDM. To create a manual BlockDevice, follow the steps below:\n\n1. Download the sample block device custom resource YAML file. \n ```\n wget\n https://raw.githubusercontent.com/openebs/node-disk-manager/master/deploy/crds/openebs_v1alpha1_blockdevice_cr.yaml\n ```\n\n2.  Edit the file and fill in the details of the blockdevice. Fields marked with optional are not mandatory and can be removed. All other fields are required and information provided will be used while claiming.\n ```\n apiVersion: openebs.io/v1alpha1\n kind: BlockDevice\n metadata:\n name: example-blockdevice\n labels:\n kubernetes.io/hostname: # like gke-openebs-user-default-pool-044afcb8-bmc0\n ndm.io/managed: \"false\" # for manual disk creation put false\n ndm.io/blockdevice-type: blockdevice\n status:\n claimState: Unclaimed\n state: Active\n spec:\n capacity:\n storage: #like 53687091200\n details:\n firmwareRevision: #optional\n model: # like PersistentDisk, optional\n serial: # like google-disk-2, optional\n compliance: #like \"SPC-4\", optional\n vendor: #like Google, optional\n devlinks:\n - kind: by-id\n links:\n - # like /dev/disk/by-id/scsi-0Google_PersistentDisk_disk-2\n - # like /dev/disk/by-id/google-disk-2\n - kind: by-path\n links:\n - # like /dev/disk/by-path/virtio-pci-0000:00:03.0-scsi-0:0:2:0\n nodeAttributes:\n nodeName: # output of `kubectl get nodes` can be used\n path: # like /dev/sdb\n ```\n3.  Apply the YAML file.\n ```\n kubectl apply -f openebs_v1alpha1_blockdevice_cr.yaml\n ```\n\nThe BlockDevice CR will be created and is then used by NDM Operator for claiming, but it won’t be managed by NDM Daemon for any changes that happen on the device. However, all the Claim/Unclaim operations and cleanup operations will be performed on this BlockDevice.\n\nPlease provide your valuable feedback & comments below and let me know what I can cover in my next blog.\n\nThis blog was originally published on Oct 22nd, 2019, on the [MayaData blog](https://blog.mayadata.io/openebs/creating-manual-blockdevice).\n","notHasFeatureImage":false,"slug":"creating-manual-blockdevice"},{"id":38,"title":"OpenEBS Node Device Management (NDM) — Troubleshooting tips","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"08-01-2020","tags":["Docker","OpenEBS","Uncategorized","Troubleshooting","Tutorials"],"excerpt":"OpenEBS Node Device Management (aka NDM) helps in discovering the block devices attached to Kubernetes nodes. In this blog, I will walk through some of the scenarios I have seen working with users on the OpenEBS Slack Channel.","content":"\nOpenEBS Node Device Management (aka NDM) helps in discovering the block devices attached to Kubernetes nodes. For many clusters, the default configuration of NDM suffices, however there are some cases where further customizations are required.\n\n> In this blog, I will walk through some of the scenarios I have seen working with Users on the [OpenEBS Slack Channel](http://slack.openebs.io/)\n\n---\n\n### NDM Quick Overview\n\nFor setting up NDM in secure mode, please see my previous [blog](https://blog.mayadata.io/openebs/configuring-openebs-to-run-with-security-enhanced-linux), and you can learn how NDM works [here](/docs/user-guides/ndm). Here is a quick snapshot of the key components of NDM.\n\n- NDM components are installed in the OpenEBS Namespace. Ensure that NDM pods part of the NDM DaemonSet are running on all the storage nodes. NDM Operator helps with allocating Block Devices to Block Device Claims and should be running.\n- NDM DaemonSet pod discovers all the block devices attached to the node and creates BlockDevice custom resource for each device. Note that NDM will filter out some of the devices like loopback device and so forth as configured in the NDM ConfigMap. `kubectl get bd -n openebs`\n- NDM creates a special type of devices called sparse devices depending on the `SPARSE_FILE_COUNT` and `SPARSE_FILE_SIZE` passed to the NDM Daemon. These devices are used in cases where nodes do not have any additional devices attached to the node and users would like to run their applications by carving out some spaces from the OS disk. The creation of sparse devices is disabled by default from OpenEBS 1.3.\n- Users or Operators like cStor Operator, Local PV provisioner interact with NDM by creating a BlockDeviceClaim CR. The BlockDeviceClaim will have properties like nodeName, required Capacity etc., The NDM operator will match these properties with the available BlockDevices and associate the one that matches all the requirements to BlockDeviceClaim.\n\n---\n\n### NDM Known Issues / Future Development Items\n\n- BlockDevices are not created for Partitions and LVM devices. If you need to use them, you have to manually create BlockDevice CR. The steps are mentioned in this [blog](https://blog.mayadata.io/openebs/creating-manual-blockdevice).\n\nOK. Let us get started with some common issues reported and how to troubleshoot them.\n\n---\n\n#### Scenario #1\n\n**BlockDevice CR is not created for a device available on my node.**\n\nI have some disks attached to the node. Installed OpenEBS, but blockdevice resources are not created for the devices.\n\n**Symptom:** I have some disks attached to the node. Installed OpenEBS, but blockdevice resources are not created for the devices.\n\n**Troubleshooting:**\n\n1. Check `lsblk` output of the node\n2. Get the NDM config map.\n3. Check if the mount point of the disk is excluded in the filter configurations in configmap.\n4. From lsblk output check if the blockdevice you want to use is an LVM/software raid/ partition/LUKS filesystem. NDM currently does not support these types.\n5. If none of the above works, the logs of NDM daemonset can be checked. It will have information of disk being detected, and at what point the disk was excluded from blockdevice creation, (like `excluded by path-filter`)\n\n ![blockdevice creation](https://cdn-images-1.medium.com/max/800/0*q8rBQFw284gRYqjg)\n\n**Resolution:** Update the filter configuration in configmap and restart the NDM DaemonSet pod. This will create the blockdevices.\n\n---\n\n#### Scenario #2\n\n**After node reboot, one blockdevice became inactive and another blockdevice was created.**\n\n**Symptom:** When a node in the cluster is rebooted, A blockdevice resource on that node was marked as inactive and a new resource was created. The new blockdevice also has the same details as the old one.\n\n**Troubleshooting:**\n\n1. Check `lsblk` output of the node\n2. Get the yaml of both blockdevices and compare them.\n3. Check to see `spec.Path` is different in both outputs.\n4. If yes, then the new blockdevice resource was created because the path changed\n5. Check if `kubernetes.io/hostname` is different, if yes, then the blockdevice was created because the hostname of the node changed.\n\n**Resolution:** If using cStor, the newly generated BD can be added in both SPC and CSP instead of the old BD resource. Thus the storage engine will claim the new BD resource and start using it.\n\n**Root Cause:** Whenever the NDM deamonset pods shutdown, all the devices on that node will be marked into an Unknown state. When the pod comes backup, all the devices on that node are marked as inactive, and then individual devices are processed for their statuses.\n\nNDM uses an md5 sum of WWN+Model+Serial of the disk to create its unique name. If none of these fields are available then NDM uses device path and hostname to create the blockdevice. There are chances that the device path/hostname has changed after reboot. If the path/hostname changes a new blockdevice resource will be created, and the old one will still be in the inactive state.\n\n---\n\n#### Scenario #3\n\n**BlockDevices are created for already used disks in which OS is installed**\n\n**Symptom:** NDM created blockdevice resources for disks which are already used for OS partitions. By default NDM excludes the blockdevices that are mounted at `/, /boot, /etc/hosts`. If these mount points are on an LVM or SoftRaid, NDM will not be able to identify that.\n\n**Resolution:** Support for LVM and software RAID is in the design phase. Once it is supported the issue will be resolved.\n\n---\n\n#### Scenario #4\n\n**Only one Blockdevice is created, when devices are connected in multipath configuration**\n\n**Symptom:** There is a disk attached in multipath configuration to a node. i.e both sdb & sdc are the same devices. But blockdevice resource is created only for sdc.\n\n**Resolution:** Support for detecting disks in multipath configuration and attaching the same disk to multiple nodes will be available in the future versions of NDM\n\n**Root Cause:** NDM generates the UID for disk identification using the disk details like WWN, Serial, etc that are fetched from the disk. In case of a disk attached in multipath configuration, the details from both sdb and sdc will be the same. Therefore, NDM will first create a blockdevice for sdb, and then moves on to create for `sdc`. But at this stage, it will find that a blockdevice with that UID already exists and will update the blockdevice information with the new path `sdc`. This results in a blockdevice existing only for sdc.\n\n---\n\n#### Scenario #5\n\n**Only single BlockDevice resource is created in a multi-node Kubernetes cluster on GKE.**\n\n**Symptom:** On a multinode kubernetes cluster in GKE, with an external GPD attached to each node. NDM is creating only one blockdevice resource, instead of one blockdevice resource per node.\n\n**Troubleshooting:**\n\n1. Was the GPD added using the gcloud CLI or google cloud console web UI?\n2. If the disk was added using gcloud CLI, check whether the ` — device-name` flag was specified during attaching the disk.\n\n**Resolution:** The command to add disk using gcloud CLI should be\n\n gcloud compute instances attach-disk --disk=disk-1 **--device-name=disk-1**\n\n**Root Cause:** gcloud CLI uses the value provided in the `device-name` flag as the serial number of the GPD when it is attached to the node. If it is left blank, Google will assign a default serial number that is unique only to the node. When multiple nodes are present, and NDM generates the UID for the blockdevice, the disks on both nodes will have the same serial number and thus the same UID.\n\nNDM from one node will create the blockdevice resource and when the other NDM daemon tries to create the resource, it finds that a resource already exists and just updates the resource.\n\nThis blog was originally published on Jan 7th, 2020 on [MayaData blog](https://blog.mayadata.io/openebs/openebs-node-device-management-ndm-troubleshooting-tips).\n","notHasFeatureImage":false,"slug":"openebs-node-device-management-(ndm) — troubleshooting-tips"},{"id":39,"title":"cStor Pool Operations via CSPC in OpenEBS","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"03-01-2020","tags":["Container Attached Storage","Cspc","Kubernetes","OpenEBS","Operations"],"excerpt":"CStor Data Engine is popular for workloads needing efficient capacity management, replicas, point in time snapshots, incremental backups, etc. The tutorial will provision striped cStor pools and perform operations on them.","content":"\n**An enhanced schema for OpenEBS cStor Pool Management**\n\nCStor Data Engine is popular for workloads needing efficient capacity management, replicas, point in time snapshots, incremental backups, etc. Since achieving Beta last year, cStor Data Engine has been powering many Stateful Applications, including a variety of databases.\n\nWhile users love cStor for its data management capabilities, they have been providing feedback that it is not as easy to use as Jiva. We have started working on user feedback starting with OpenEBS 1.2 release and incrementally enhancing the cStor functionality.\n\nAs cStor is already running in production, the new changes are being introduced via a set of new cStor Custom Resources, which will get into in a bit. The users can continue to use the current schema, as we stabilize the new design and support a seamless migration. You can help us improve by providing feedback on the new design [here](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/cstor-operator) or by raising [issues](https://github.com/openebs/openebs/issues).\n\nTo give a high-level overview of the new design, the following will be changed:\n\n- A new component/deployment called cspc-operator will be deployed for managing the new custom resources.\n- CSPC — cStor Pool Cluster will be replacing the SPC. The spec of the CSPC has been modified to provide cleaner abstractions for managing blockdevices on each node.\n- CSPI — cStor Pool Instance will be replacing CSP.\n\nThe new schema will only support manual cStor Pools creation as we learned that supporting both manual and auto using a single customer CR can lead to more confusion. In future releases, a new schema will be introduced to support the automatic creation of the cStor Pools.\n\nCSPC can be used to provision cStor pool as well as carry out day 2 pool operations such as: Following storage day 2 operations are supported via the CSPC schema:\n\n- Pool Expansion ( Supported in OpenEBS version >=1.2, alpha )\n- Pool Deletion ( Supported in OpenEBS version >=1.2, alpha )\n- Pool Scale Up ( Supported in OpenEBS version >=1.2, alpha )\n- Block Device Replacement ( Supported in OpenEBS version >=1.5, alpha)\n\nLet us go through a short hands-on tutorial to understand these.\n\nThe tutorial will provision striped cStor pools and perform operations on them (this can be done for other cStor raid groups too and I will post that in a separate blog).\n\n#### Prerequisite Steps\n\n- Kubernetes cluster of version >= 1.14 with 3 worker nodes.\n- Attach at least 2 disks to each of the nodes to follow up with the tutorial. I am using GKE and steps to create and attach a disk to a node are given in the following link :\n[https://cloud.google.com/sdk/gcloud/reference/compute/disks/create\n](https://cloud.google.com/sdk/gcloud/reference/compute/disks/create)[https://cloud.google.com/sdk/gcloud/reference/compute/instances/attach-disk\n](https://cloud.google.com/sdk/gcloud/reference/compute/instances/attach-disk)If you are using other providers, check with their reference manuals on how to attach a disk. Also, feel free to reach out on the OpenEBS slack channel if you need any assistance.\n- I have used the following script to create 6 disks.\n *{ for i in {1..6}; do gcloud compute disks create demo-disk-$i — zone=us-central1-a — size=100GB; done; }*\n- Now, I will attach 2 disks to each of the nodes. I have used the following commands to attach.\n\n *for i in {1..2}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41–2ldq — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n *for i in {3..4}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41-bb69 — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n *for i in {5..6}; do gcloud compute instances attach-disk gke-cstor-demo-default-pool-3385ab41-hrmr — disk demo-disk-$i — device-name demo-disk-$i — zone us-central1-a; done*\n\n- Install OpenEBS 1.5. Run following to install: (Note that block device replacement support starts from OpenEBS version 1.5. There will be a separate blog post to describe that.)\n ```\n kubectl apply -f [https://openebs.github.io/charts/openebs-operator-1.5.0.yaml](https://openebs.github.io/charts/openebs-operator-1.6.0.yaml)\n ```\n- Install CStor-Operator. Run following to install:\n ```\n kubectl apply -f [https://raw.githubusercontent.com/openebs/openebs/master/k8s/cstor-operator.yaml](https://raw.githubusercontent.com/openebs/openebs/master/k8s/cstor-operator.yaml)\n ```\n\n#### Pool Provisioning\n\nYou need to specify cStor pool intent in a CSPC YAML to provision cStor pools on nodes. I am going to provision 3 stripe cStor pools. Let us prepare a CSPC YAML now.\n\nFollowing command list all block devices which represent your attached disks. I am going to pick 1 block device from each node to form a CSPC YAML.\n\n $ kubectl get bd -n openebs\n NAME NODENAME SIZE CLAIMSTATE STATUS AGE\n\n blockdevice-474d20376a541a8fb372d44f5bc361ea gke-cstor-demo-default-pool-3385ab41-hrmr 107374182400 Unclaimed Active 34s\n\n blockdevice-9773ccb731e4e3e10c2838411f5f8b2a gke-cstor-demo-default-pool-3385ab41-bb69 107374182400 Unclaimed Active 37s\n\n blockdevice-9c8df120e17379bfd1fe5c3ce9aa8185 gke-cstor-demo-default-pool-3385ab41-bb69 107374182400 Unclaimed Active 37s\n\n blockdevice-ada8ef910929513c1ad650c08fbe3f36 gke-cstor-demo-default-pool-3385ab41-2ldq 107374182400 Unclaimed Active 34s\n\n blockdevice-d2d59218ed78560b206143ab0641470c gke-cstor-demo-default-pool-3385ab41-hrmr 107374182400 Unclaimed Active 34s\n\n blockdevice-f6408e135943e1ee45171034655a8b88 gke-cstor-demo-default-pool-3385ab41-2ldq 107374182400 Unclaimed Active 34s\n\nMy CSPC YAML is on the following link (you can copy/download and make changes accordingly).\n\n[https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml](https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml)\n\nSave the above file with your changes and apply the above YAML to provision cStor pools.\n\n $ kubectl apply -f https://raw.githubusercontent.com/openebs/elves/a8ce5d6401f1ab829a35214ea01c284ccfb03c13/demo/cspc/cspc-stripe.yaml\n cstorpoolcluster.openebs.io/cspc-stripe created\n\n $ kubectl get cspi -n openebs\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-8vtx gke-cstor-demo-default-pool-3385ab41-2ldq 69.5K 99.5G 99.5G ONLINE 99s\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 69.5K 99.5G 99.5G ONLINE 99s\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 69.5K 99.5G 99.5G ONLINE 99s\n\nIf you want to deploy a workload to use the cStor pool, go through the following cStor CSI driver link:\n[https://github.com/openebs/cstor-csi/blob/master/README.md](https://github.com/openebs/cstor-csi/blob/master/README.md)\n\nIn the next section, we will do pool expansion.\n\n#### Pool Expansion\n\nLet us expand one cStor stripe pool on a node by editing the CSPC cspc-stripe.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n\nSimply, add one more block device as follows. Make sure you do not put a block device that is a part of any other CSPC. I have added block device *blockdevice-d2d59218ed78560b206143ab0641470c* in my case.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n\n ...\n spec:\n pools:\n - nodeSelector:\n kubernetes.io/hostname: gke-cstor-demo-default-pool-3385ab41-hrmr\n poolConfig:\n cacheFile: \"\"\n compression: \"off\"\n defaultRaidGroupType: stripe\n overProvisioning: false\n raidGroups:\n - blockDevices:\n - blockDeviceName: blockdevice-474d20376a541a8fb372d44f5bc361ea\n capacity: \"\"\n devLink: \"\"\n # Block device added. This must be attached to node\n # gke-cstor-demo-default-pool-3385ab41-hrmr and should not be used by any other CSPC.\n - blockDeviceName: blockdevice-d2d59218ed78560b206143ab0641470c\n\n isReadCache: false\n isSpare: false\n isWriteCache: false\n type: stripe\n ...\n\nSave the above changes.\n\n $ kubectl get cspi -n openebs\n\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-8vtx gke-cstor-demo-default-pool-3385ab41-2ldq 268K 99.5G 99.5G ONLINE 17m\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 292K 99.5G 99.5G ONLINE 17m\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 232K 199G 199G ONLINE 17m\n\nYou can see that the pool cspc-stripe-x9pw got expanded. (Re-run the get cspi command if the bigger size is not shown instantaneously). Similarly, you can expand the other stripe cStor pools of the CSPC.\n\n#### Pool Deletion\n\nTo delete a cStor pool from node simple remove the node configuration from the CSPC.\n\nLet us delete one pool from CSPC cspc-stripe.\n\n $ kubectl edit cspc cspc-stripe -n openebs\n I have removed following entire config from the CSPC to delete pool on host gke-cstor-demo-default-pool-3385ab41-2ldq.\n ...\n - nodeSelector:\n kubernetes.io/hostname: gke-cstor-demo-default-pool-3385ab41-2ldq\n poolConfig:\n cacheFile: \"\"\n compression: \"off\"\n defaultRaidGroupType: stripe\n overProvisioning: false\n raidGroups:\n - blockDevices:\n - blockDeviceName: blockdevice-ada8ef910929513c1ad650c08fbe3f36\n capacity: \"\"\n devLink: \"\"\n\n isReadCache: false\n isSpare: false\n isWriteCache: false\n type: stripe\n ...\n\nAfter you are done removing, save it and you will see that pool on that node has been deleted.\n\n $ kubectl get cspi -n openebs\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 335K 99.5G 99.5G ONLINE 25m\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 372K 199G 199G ONLINE 25m\n\nYou can see that only two stripe pools are present.\n\n#### Pool Scale Up\n\nWe can even create a pool on other nodes by adding node config to the CSPC YAML. It is just the reverse of pool deletion.\n\nLet us try to add the same node config that we removed to again create the pool on the node. Make sure that while copy, pasting and editing indentation of YAML is not disturbed else error will be thrown and YAML will not be persisted.\n\nAdd the removed config from the pool deletion section to the CSPC.\n\n $ kubectl get cspi -n openebs\n NAME HOSTNAME ALLOCATED FREE CAPACITY STATUS AGE\n\n cspc-stripe-h7kl gke-cstor-demo-default-pool-3385ab41-bb69 318K 99.5G 99.5G ONLINE 30m\n\n cspc-stripe-twvv gke-cstor-demo-default-pool-3385ab41-2ldq 50K 99.5G 99.5G ONLINE 9s\n\n cspc-stripe-x9pw gke-cstor-demo-default-pool-3385ab41-hrmr 380K 199G 199G ONLINE 30m\n\nYou can see that a new pool cspc-stripe-twvv has come up online.\n\nYou can also delete the CStorPoolCluster too. If you do so, all the CStorPoolInstances associated with it will get deleted.\n\nThe command is :\n ```\n $ kubectl delete cspc \n ```\nNOTES:\n\n- Whenever a block device is used for pool creation or expansion a blockdeviceclaim CR is created and the block device will show a Claimed status.\n The following are the commands to visualize this.\n ```\n kubectl get blockdevice -n openebs \n kubectl get blockdeviceclaim -n openebs\n ```\n- Whenever a pool is deleted for a CSPC by removing the node config, the associated block-device is not ‘Unclaimed’ and if the same block device needs to be used in another CSPC, the associated blockdeviceclaim needs to be cleared manually. Although, the block-device can be again used for the same CSPC. \n\nTo unclaim a block device claim, below are the steps.\n\n\n $ kubectl get bdc -n openebs\n NAME BLOCKDEVICENAME PHASE AGE\n\n bdc-cstor-83b8e958-d978-11e9-b0e6-42010a800189 blockdevice-9773ccb731e4e3e10c2838411f5f8b2a Bound 32m\n\n bdc-cstor-8581dac0-d978-11e9-b0e6-42010a800189 blockdevice-ada8ef910929513c1ad650c08fbe3f36 Bound 32m\n\n bdc-cstor-85bddd0e-d978-11e9-b0e6-42010a800189 blockdevice-474d20376a541a8fb372d44f5bc361ea Bound 32m\n\n bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 blockdevice-d2d59218ed78560b206143ab0641470c Bound 15m\n\nLet us say, we want to unclaim block device blockdevice-d2d59218ed78560b206143ab0641470c, then we need to delete the associated blockdeviceclaim i.e. bdc-cstor-85c0fd2a-d978–11e9-b0e6–42010a800189.\n\n $ kubectl edit bdc bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 -n openebs\n Remove the finalizer \"cstorpoolcluster.openebs.io/finalizer\" by editing the bdc from above command.\n After that, execute following\n\n $ kubectl delete bdc bdc-cstor-85c0fd2a-d978-11e9-b0e6-42010a800189 -n openebs\n\nNow the device will get unclaimed. Please note the following:\nBDC CRs should be deleted only when their association with CSPC has been removed. Otherwise, data corruption can happen.\n\n#### Block Device Replacement\n\nThe CSPC operator in OpenEBS (≥1.5 version ) enables you to do the replacement of the block devices in case it has gone bad. I will follow up with another blog post that will cover this block device replacement scenario.\n\nHope that the tutorial helps in understanding the pool operations steps.\n\nIf you have any questions or face any problems, feel free to reach out to me on OpenEBS slack channel.\n\nThank You!\n\nThis blog was originally published on Dec 13, 2019 on [MayaData blog](https://blog.mayadata.io/openebs/cstor-pool-operations-via-cspc-in-openebs).\n","notHasFeatureImage":false,"slug":"cstor-pool-operations-via-cspc-in-openebs"},{"id":40,"title":"OpenEBS Dynamic Volume Provisioning on ZFS","author":"Pawan Prakash Sharma","author_info":"It's been an amazing experience in Software Engineering because of my love for coding. In my free time, I read books, play table tennis and watch tv series","date":"13-12-2019","tags":["CNCF","Kubernetes","Persistent Volume","ZFS","OpenEBS"],"excerpt":"OpenEBS’ ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. This blog will demonstrate how to deploy a Percona application on the ZFS storage system with OpenEBS.","content":"\nOpenEBS’ ZFS driver binds a ZFS file system into the Kubernetes environment and allows users to provision and de-provision volumes dynamically. This blog will demonstrate how to deploy a Percona application on the ZFS storage system with OpenEBS.\n\nUsing a ZFS Local PV has the following advantages — as opposed to Kubernetes native Local PV backed by direct attached devices:\n\n- Sharing of the devices among multiple application pods.\n- Enforcing quota on the volumes, making sure the pods don’t consume more than the capacity allocated to them.\n- Ability to take snapshots of the Local PV\n- Ability to sustain the disk failures — using the ZPOOL RAID functionality\n- Ability to use data services like compression and encryption.\n\nIn this post, I will demonstrate how we can use ZFS Local PV for deploying a Percona application.\n\n**Setup**\n\nWe will be using GKE with Kubernetes 1.14+ version with Ubuntu 18.4 installed on each node. We have to set up the node with ZFS utility once the cluster is up and running. We need to install [zfsutils-linux](https://packages.ubuntu.com/bionic/zfsutils-linux) on each node and use [ZFS commands](https://www.thegeekdiary.com/solaris-zfs-command-line-reference-cheat-sheet/) to set up a storage pool.\n\n $ apt-get install -y zfsutils-linux\n\nCreate and attach the disk (if not already attached) to the nodes for setting up the ZPOOL:\n\n $ gcloud compute disks create --size --type pd-standard --zone us-central1-a\n\n $ gcloud compute instances attach-disk --disk --zone us-central1-a\n\nCreate the zpool on each node using the attached disks (sdb and sdc):\n\n $ zpool create zfspv-pool mirror /dev/sdb /dev/sdc\n\nHere we are creating a mirrored ZPOOL, we can create any kind of pool as per our requirement (raidz, striped or mirror).\n\nCheck the zpool list:\n\n $ zfs list\n NAME USED AVAIL REFER MOUNTPOINT\n\n zfspv-pool 644K 96.4G 176K /zfspv-pool\n\nInstall OpenEBS ZFS driver :\n\n $ kubectl apply -f [https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml](https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml)\n\nCheck that the driver is installed:\n\n $ kubectl get pods -n kube-system -l role=openebs-zfs\n\n NAME READY STATUS RESTARTS AGE\n\n openebs-zfs-controller-0 4/4 Running 0 5h28m\n\n openebs-zfs-node-4d94n 2/2 Running 0 5h28m\n\n openebs-zfs-node-gssh8 2/2 Running 0 5h28m\n\n openebs-zfs-node-twmx8 2/2 Running 0 5h28m\n\n**Create The Storage Class:**\n\n $ cat sc.yaml\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: percona-sc\n allowVolumeExpansion: true\n parameters:\n poolname: \"zfspv-pool\"\n provisioner: zfs.csi.openebs.io\n\nThe storage class has a *poolname* parameter, which means that any volume provisioned using this storage class will be provisioned in this pool (zfspv-pool here). The provisioner *zfs.csi.openebs.io* is the provisioner name for the ZFS driver. You can change the poolname to the name of the ZPOOL which you have created for the provisioning. Apply the YAML to create the storage class:\n\n $ kubectl apply -f sc.yaml\n\n storageclass.storage.k8s.io/percona-sc created\n\n**Create The PVC:**\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: percona-pvc\n spec:\n storageClassName: percona-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 4Gi\n\nCreate the PVC using the storage class created for the ZFS driver. You can request for the storage space via storage parameter as shown in above PVC for putting a request for 4Gi storage. Apply the YAML to create the PVC request.\n\n $ kubectl apply -f pvc.yaml\n\n persistentvolumeclaim/percona-pvc\n\nCheck the PVC\n\n $ kubectl get pvc\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n\n percona-pvc Bound pvc-ecdb16e2-e03a-11e9-b418–42010a80006b 4Gi RWO percona-sc 5m39s\n\nHere, we can see that the PVC has been created and bound also as well.\n\n**Percona YAML:**\n\n apiVersion: v1\n kind: ConfigMap\n metadata:\n annotations:\n name: sqltest\n namespace: default\n data:\n sql-test.sh: |\n #!/bin/bash\n\n DB_PREFIX=\"Inventory\"\n DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`\n DB_NAME=\"${DB_PREFIX}_${DB_SUFFIX}\"\n\n echo -e \"nWaiting for mysql server to start accepting connections..\"\n retries=10;wait_retry=30\n for i in `seq 1 $retries`; do\n mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1\n rc=$?\n [ $rc -eq 0 ] && break\n sleep $wait_retry\n done\n\n if [ $rc -ne 0 ];\n then\n echo -e \"nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exitingn\"\n exit 1\n fi\n mysql -uroot -pk8sDem0 -e \"CREATE DATABASE $DB_NAME;\"\n mysql -uroot -pk8sDem0 -e \"CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"INSERT INTO Hardware (id, name, owner, description) values (1, \"dellserver\", \"basavaraj\", \"controller\");\" $DB_NAME\n mysql -uroot -pk8sDem0 -e \"DROP DATABASE $DB_NAME;\"\n ---\n apiVersion: apps/v1\n kind: Deployment\n metadata:\n name: percona\n labels:\n name: percona\n spec:\n replicas: 1\n selector:\n matchLabels:\n name: percona\n template:\n metadata:\n labels:\n name: percona\n spec:\n containers:\n - resources:\n name: percona\n image: openebs/tests-custom-percona:latest\n imagePullPolicy: IfNotPresent\n args:\n - \"--ignore-db-dir\"\n - \"lost+found\"\n env:\n - name: MYSQL_ROOT_PASSWORD\n value: k8sDem0\n ports:\n - containerPort: 3306\n name: percona\n volumeMounts:\n - mountPath: /var/lib/mysql\n name: demo-vol1\n - mountPath: /sql-test.sh\n subPath: sql-test.sh\n name: sqltest-configmap\n livenessProbe:\n exec:\n command: [\"bash\", \"sql-test.sh\"]\n initialDelaySeconds: 30\n periodSeconds: 1\n timeoutSeconds: 10\n volumes:\n - name: demo-vol1\n persistentVolumeClaim:\n claimName: percona-pvc\n - name: sqltest-configmap\n configMap:\n name: sqltest\n\n ---\n apiVersion: v1\n kind: Service\n metadata:\n name: percona-mysql\n labels:\n name: percona-mysql\n spec:\n ports:\n - port: 3306\n targetPort: 3306\n selector:\n name: percona\n\nApply the configuration:\n\n $ kubectl apply -f percona.yaml\n\n configmap/sqltest created\n deployment.apps/percona created\n service/percona-mysql created\n\nCheck the status of the Pod.\n\n $ kubectl get po\n\n NAME READY STATUS RESTARTS AGE\n\n percona-7456dc6f7b-nnfcz 1/1 Running 0 67s\n\nWe can go the node where percona pod is scheduled and see that a volume has been created in the pool zfspv-pool using the ZFS list command,:-\n\n $ zfs list\n\n NAME USED AVAIL REFER MOUNTPOINT\n\n zfspv-pool 115M 96.3G 176K /zfspv-pool\n\n zfspv-pool/pvc-ecdb16e2-e03a-11e9-b418–42010a80006b 102M 96.3G 102M -\n\n### Summary\n\nAs demonstrated in this blog, OpenEBS makes it easy for the Kubernetes applications to take advantage of all the technical features provided by ZFS storage.\n\n### Important links\n\n[https://github.com/openebs/zfs-localpv](https://github.com/openebs/zfs-localpv)\n","notHasFeatureImage":false,"slug":"openebs-dynamic-volume-provisioning-on-zfs"},{"id":41,"title":"ECK & OpenEBS — Data Ops Streamlines Deployment","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"23-08-2019","tags":["Eck","Elasticsearch","Kubernetes","LocalPV","OpenEBS"],"excerpt":"Using OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch ECK scaling and resilience while finally delivering a completely declaratively-managed application stack.","content":"\n### TL;DR\n\nUsing OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch ECK scaling and resilience while finally delivering a completely declaratively-managed application stack. Let’s review how.\n\n**Good News: The recently shipped** [**Elastic Cloud on Kubernetes (ECK)**](https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond) delivers Elasticsearch clusters as native, distributed Kubernetes Operators. Elasticsearch is a distributed, open source search and analytics engine for all types of data. Widely used, Elasticsearch is storage-intensive because it builds an inverted index of collections of JSON objects that are related to each other to allow very fast full-text searches. The result is a simplified deployment of ElasticSearch for the Kubernetes admins or SREs as well as a simplified developer experience.\n\n**Bad News: ElasticSearch uses fast local storage but it does not address storage provisioning and management.** ECK use of the [static provisioner for Local](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner) PV requires administrators to manually format, mount, and configure disks on Kubernetes nodes. This is a PITA, as they say.\n\n**Good News: OpenEBS removes the burden of Storage Operations for ECK Deployments.** With OpenEBS, administrators can easily manage local PV storage resources for ECK seamlessly across both on-premises and multiple clouds, simplifying ElasticSearch scaling and resilience.\n\n### Vanilla K8s (PITA)\n\nTypically, ElasticSearch is deployed one of two ways:\n\n- **Dedicated mode:** Elastic pods are using LocalPV which are real disks, and they need to be dynamically provisioned as the pods’ scale.\n- **Shared mode:** Elastic pods are using LocalPVs from shared storage for better capacity economics.\n\n Here is the detail on K8s static provisioners from [K8s documentation](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner)\n\n *Note that the local storage provisioner is different from most provisioners and does not support dynamic provisioning. Instead, it requires that administrators preconfigure the local volumes on each node and if volumes are supposed to be*\n\n- *Filesystem volumeMode (default) PVs — mount them under discovery directories.*\n- *Block volumeMode PVs — create a symbolic link under discovery directory to the block device on the node.*\n\n *The provisioner will manage the volumes under the discovery directories by creating and cleaning up PersistentVolumes for each volume.*\n\n *This means a Kubernetes administrator must manually manage storage operations outside of the ECK operator itself. Specifically,*\n\n *– Capacity resize of underlying volumes.*\n\n *– Capacity management of shared storage when deployed in shared mode.*\n\n *– Shifting of some of the data volumes from one node to another automatically in case a node has to be cordoned or drained.*\n\n *– Move data to other Kubernetes clusters.*\n\n**Note: insert administrator sweat equity here^.**\n\n### OpenEBS LocalPV + Data Ops (Good)\n\nAn OpenEBS storage cluster, itself Kubernetes-native, simplifies and automates storage provisioning and management operations either on your data center or in the cloud (or spanning both!). OpenEBS provisioners use OpenEBS disk pool operators and built-in data management capabilities to dynamically provision LocalPVs in a host path or in a disk mode to ECK pods.\n\n![Multiple elastic data nodes sharing a pool of disks for thim provisioning effect](https://cdn-images-1.medium.com/max/800/1*PHw4zrcvJF_w-VcTI90RbA.png)\n\n### K8s Advantage: Declarative Data Plane\n\nOpenEBS uses a Declarative Data Plane to manage storage operations which aligns architecturally with Kubernetes Operators generally, and specifically with the ECK operator. Storage is typically the last “architectural mile” of Kubernetes deployments. Storage has gravity for applications that tends to tie Kubernetes clusters to the storage infrastructure beneath it. The OpenEBS Declarative Data Plane removes that last architectural mile by providing a completely Kubernetes-native software-defined storage infrastructure that spans on-premise and cloud resources and lets administrators manage Kubernetes application gravity consistently across all sites.\n\n**OpenEBS-managed storage means that the end-to-end operations of the entire Kubernetes application stack is finally managed, top to bottom, in a completely declarative way.**\n\n### Configuring a Dynamic localPV for ECK\n\nThe StorageClass spec for [OpenEBS LocalPV](/docs/concepts/localpv) for automatically choosing an available disk on the node and mounting that disk with ext4 volume would look like the following:\n\n cat <> ~/.ssh/authorized_keys' \n eval 'ssh-agent'\n \n - name: Copying the SSH key into compute nodes\n shell: |\n ssh -o StrictHostKeyChecking=no root@{{item}} 'echo >> ~/.ssh/authorized_keys'\n eval 'ssh-agent' \n with_items: \"\"\n \n - name: SSH from master to each Nodes \n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@{{item}} ls'\n with_lines: cat ./ip.csv\n \n - name: Generating random number\n shell: date +%s\n register: rand_num\n \n - name: Setting up the master hostname \n shell: |\n ssh -o StrictHostKeyChecking=no root@ 'echo master-. > /etc/hostname'\n ssh -o StrictHostKeyChecking=no root@ 'systemctl start systemd-hostnamed'\n \n - name: Setting up the compute nodes hostname\n shell: |\n ssh -o StrictHostKeyChecking=no root@ 'echo node-. > /etc/hostname'\n ssh -o StrictHostKeyChecking=no root@ 'systemctl start systemd-hostnamed' \n with_together: \n - [ '1', '2', '3' ]\n - \"\" \n \n #The VMs are already subscribed with some credentials. Need to unsubscribe the VMs & subscribe it again with new credentials.\n - name: Unsubscribing the nodes\n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager unregister'\n ignore_errors: true\n with_lines: cat ./ip.csv \n \n - name: Subscribing the nodes\n shell: |\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager register --username= --password='\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager attach --auto'\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager refresh'\n with_lines: cat ./ip.csv\n \n - name: Getting the pool-id\n shell: ssh -o StrictHostKeyChecking=no root@ 'subscription-manager list --available --matches '*OpenShift*' | grep \"Pool ID\" | awk '\\''NR == 1'\\'' | awk '\\''{print $3}'\\'''\n register: pool_id\n \n - name: Attaching pool to each nodes\n shell: | \n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager attach --pool='\n ssh -o StrictHostKeyChecking=no root@{{item}} 'subscription-manager repos --enable=\"rhel-7-server-rpms\" --enable=\"rhel-7-server-extras-rpms\" --enable=\"rhel-7-server-ose-3.10-rpms\" --enable=\"rhel-7-server-ansible-2.4-rpms\"'\n with_lines: cat ./ip.csv\n \n - name: Joining the nodes to the DNS Server\n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'echo | realm join --user=Administrator '\n with_lines: cat ./ip.csv\n \n - name: SSH from master to master using DNS\n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@master-. ls' \n \n - name: SSH from master to compute nodes using DNS\n shell: ssh -o StrictHostKeyChecking=no root@ 'ssh -o StrictHostKeyChecking=no root@node{{item}}-. ls'\n with_items: \n - 1\n - 2\n - 3 \n \n - name: Replacing master ip in inventory\n replace:\n path: ./inventory.yml\n regexp: \"master_ip\"\n replace: \"\" \n \n - name: Replace compute node ip in inventory\n replace:\n path: ./inventory.yml\n regexp: \"\"\n replace: \"\"\n with_together: \n - [ 'node1_ip', 'node2_ip', 'node3_ip' ]\n - \"\" \n \n - name: Replacing master DNS in inventory\n replace:\n path: ./inventory.yml\n regexp: \"master_dns\"\n replace: \"master-.\"\n \n - name: Replacing nodes DNS in inventory\n replace:\n path: ./inventory.yml\n regexp: \"\"\n replace: \"\"\n with_together:\n - [ 'node1_dns', 'node2_dns', 'node3_dns' ]\n - [ 'node1-.', 'node2-.', 'node3-.' ] \n \n - name: Copying inventory into master\n shell: scp -o StrictHostKeyChecking=no inventory.yml root@:/root/openshift-ansible/inventory/\n \n - name: Checking out to release branch-3.10\n shell: ssh -o StrictHostKeyChecking=no root@ 'cd /root/openshift-ansible && git checkout release-3.10'\n \n - name: Running Openshift pre-requisites\n shell: ssh -o StrictHostKeyChecking=no root@ 'ansible-playbook -i /root/openshift-ansible/inventory/inventory.yml /root/openshift-ansible/playbooks/prerequisites.yml -vv'\n \n - name: Deploying openshift cluster\n shell: ssh -o StrictHostKeyChecking=no root@ 'ansible-playbook -i /root/openshift-ansible/inventory/inventory.yml /root/openshift-ansible/playbooks/deploy_cluster.yml -vv'\n \n - name: Disabling selinux on each nodes \n shell: ssh -o StrictHostKeyChecking=no root@{{item}} 'setenforce 0'\n with_lines: cat ./ip.csv \n \n - name: Copying bash file in master & Executing\n shell: | \n scp -o StrictHostKeyChecking=no post_install_setting.sh root@:/root/ \n ssh -o StrictHostKeyChecking=no root@ 'bash post_install_setting.sh && rm post_install_setting.sh'\n\n### App deployers\n\nLitmus job for deploying NuoDB EE using OpenEBS volumes for database requirements.\n\nPrerequisites for running this litmus job is to have NuoDB Enterprise edition subscription and have docker images available in the cluster.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/OpenShift/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/deployers/OpenShift/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: litmus-nuodb-\n namespace: litmus\n spec:\n template:\n metadata:\n name: litmus\n labels:\n app: nuodb-deployment\n \n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n \n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays, actionable, default\n value: default\n \n - name: PROVIDER_STORAGE_CLASS\n # Supported values: openebs-standard, local-storage, openebs-standalone\n value: openebs-cstor-sparse\n \n - name: NUODB_VERSION\n value: ee\n \n - name: APP_PVC\n value: demo-vol-claim\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n # Application label\n - name: APP_LABEL\n value: 'app=nuodb'\n \n # Use 'deprovision' for app-clean up\n - name: ACTION\n value: provision\n \n # Set THP to disable in case platform is AWS or OpenShift\n - name: THP\n value: disable\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/deployers/OpenShift/test.yml -i /etc/ansible/hosts -v; exit 0\"]\n\n### Chaos jobs — NuoDB\n\nLitmus job for inducing chaos on NuoDB application components such as Admin, Storage Manager and Transaction Engine.\n\nFor inducing various components chaos, user has to pass application component specific label as the Job Env to the litmus book.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/chaos/app_pod_failure/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/apps/nuodb/chaos/app_pod_failure/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: nuodb-app-chaos-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: nuodb-app-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: DEPLOY_TYPE\n value: statefulset\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/chaos/app_pod_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Networking\n\nLitmus job for inducing network delays between application and openebs target pod.\n\n[https://raw.githubusercontent.com/openebs/litmus/master/experiments/chaos/openebs_target_network_delay/run_litmus_test.yml](https://raw.githubusercontent.com/openebs/litmus/master/experiments/chaos/openebs_target_network_delay/run_litmus_test.yml)\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: nuodb-app-chaos-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: nuodb-app-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: DEPLOY_TYPE\n value: statefulset\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./nuodb/chaos/app_pod_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Storage\n\nLitmus job for inducing OpenEBS cStor storage target kill and verify the application availability.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-target-failure-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: openebs-target-failure\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n - name: TARGET_NAMESPACE\n value: openebs\n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: DATA_PERSISTENCY\n value: \"\" \n \n # CHOS_TYPE values : target-zrepl-kill , target-kill , target-delete \n - name: CHAOS_TYPE\n value: \"target-zrepl-kill\"\n \n # TARGET_CONTAINER values: cstor-volume-mgmt , cstor-istgt\n - name: TARGET_CONTAINER\n value: \"cstor-volume-mgmt\"\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_target_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\nLitmus job for inducing OpenEBS cStor storage pool kill and verify the application availability.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-pool-failure-\n namespace: litmus\n spec:\n template:\n metadata:\n labels:\n name: openebs-pool-failure\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n \n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n env:\n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: APP_NAMESPACE\n value: nuodbns\n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: DATA_PERSISTENCY\n value: \"\" \n \n - name: CHAOS_TYPE\n value: \"pool-kill\"\n \n - name: CHAOS_ITERATIONS\n value: \"2\" \n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_pool_failure/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n\n### Chaos jobs — Kubernetes\n\nLitmus job for inducing kubelet/docker service crash and verify impact on the application running on the node.\n\nNote: This litmus job is specific to AWS platform.\n\n ---\n apiVersion: batch/v1\n kind: Job\n metadata:\n generateName: openebs-app-svc-chaos-\n namespace: litmus \n spec:\n template:\n metadata:\n labels:\n name: openebs-app-svc-chaos\n spec:\n serviceAccountName: litmus\n restartPolicy: Never\n \n #nodeSelector:\n # kubernetes.io/hostname:\n \n containers:\n - name: ansibletest\n image: openebs/ansible-runner:ci\n imagePullPolicy: Always\n env: \n - name: ANSIBLE_STDOUT_CALLBACK\n #value: log_plays\n #value: actionable\n value: default\n \n - name: OPERATOR_NAMESPACE\n value: openebs\n \n - name: APP_NAMESPACE\n value: nuodbns \n \n - name: APP_LABEL\n value: 'nodetype=sm'\n \n - name: APP_PVC\n value: archive-sm-0\n \n # Set value to kubelet/docker \n - name: SVC_CHAOS\n value: docker\n \n - name: CHAOS_DURATION\n value: \"300\" # in seconds\n \n - name: LIVENESS_APP_LABEL\n value: \"\"\n \n - name: LIVENESS_APP_NAMESPACE\n value: \"\"\n \n - name: PLATFORM\n value: \"AWS\"\n \n command: [\"/bin/bash\"]\n args: [\"-c\", \"ansible-playbook ./percona/chaos/openebs_app_svc_chaos/test.yml -i /etc/ansible/hosts -vv; exit 0\"]\n","notHasFeatureImage":false,"slug":"using-chaos-engineering-to-harden-applications-that-use-nuodb,-openebs"},{"id":44,"title":"OpenEBS community releases v1.1, maintaining a faster release cadence.","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"08-08-2019","tags":["Kubernetes","Microservices","OpenEBS","Persistent Volume","Stateful Workloads"],"excerpt":"In this blog, I will provide a quick summary of the changes that were released as part of OpenEBS version 1.1 and also share some thoughts on the evolving project management process in OpenEBS and how it is helping to maintain a faster release cadence.","content":"\nIn this blog, I will provide a quick summary of the changes that were released as part of OpenEBS version 1.1 and also share some thoughts on the evolving project management process in OpenEBS and how it is helping to maintain a faster release cadence.\n\nOpenEBS Release 1.1 has been about fixing and documenting the cross-platform usability issues reported by users and also laying the foundation for some of the long-overdue backlogs like CSI Driver, automated upgrades, day 2 operations, and others.\n\nBefore we get into the specifics of the current release, the last three OpenEBS releases have set an interesting precedent towards attaining a *monthly release cadence*.\n\nOpenEBS was built by adopting the cloud-native and microservices principles, and it is almost only natural to also reap the benefits of true DevOps product with faster releases. It is easier said than done though! After having experimented with several tools and having looked at various open-source projects including Kubernetes, we have arrived at the following process, which is helping us maintain release cadence and thereby being responsive to the user requirements.\n\n- Responsiveness — Almost all the active contributors and maintainers of the OpenEBS project are reachable and online in the OpenEBS Community Slack. OpenEBS has been credited as being one of the most responsive CNCF community projects — and thanks to the community, OpenEBS Developers are getting feedback directly from end-users. This eliminates layers of requirements for implementation and improves the feedback loop.\n- Clarity of criteria for alpha and beta — Recently we clarified that our release gates are defined by OpenEBS Litmus based GitLab pipelines that run end-to-end tests on multiple different platforms and stateful workloads. Perhaps goes without saying — however we use these pipelines to catch any regressions. What is more — a feature is marked as Beta only after it has been added to the test pipelines. For example, LocalPV as of OpenEBS 1.1 is now Beta because it is passing these tests — and also is seeing a lot of production usage as well.\n- Backlog grooming — At the start of the release, we look at the backlogs which are on [GitHub](https://github.com/openebs/openebs/issues). Items are selected based on contributor availability and balancing the development of new features, fixing existing features, updating and improving documentation, improving e2e coverage, and hardening the usage of OpenEBS on new platforms. As an example of a new platform, we have seen quite a bit of usage of especially the low footprint Jiva on ARM and are now releasing container images for built for the ARM64 architecture, making OpenEBS operational on RPi4 as well as Amazon A1 instances or Packet’s powerful ARM Compute servers. As another example, we are hardening the use of OpenEBS for Konvoy from our friends at Day2IQ — and shortly we will see Konvoy on OpenEBS.ci. As reminder OpenEBS.ci is a public way for showing that all commits to OpenEBS master are tested against a set of workloads and platforms. OpenEBS also now appears in the [OpenShift Operator Hub](https://github.com/openebs/helm-operator/blob/master/olm/README.md) and on the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/MayaData-OpenEBS-Cloud-Native-Storage/B07TFS9Q8D) as well\n- Tracking items — The list of selected items is tracked for the current release using these [Google Sheets](https://docs.google.com/spreadsheets/d/1bbphUqbxShBhgr1VHaEQUzIGMaJJacPNKc1ckNXU1QE/). It is not fancy, but it helps to get all the collaborators together and very easily provides a no-barrier objective follow-up — between release manager, leads, and reviewers. The format of the sheet is a modified version of what is used by Kubernetes sig-storage.\n- Role of core committers — As core contributors, our responsibility is to detail the design and to list the implementation tasks — including covering the integration and upgrade tests. Each granular task is updated in the above project sheet and then we ask for help from the community to fix some of these items. The designs themselves are discussed and maintained as GitHub PRs [here](https://github.com/openebs/openebs/tree/master/contribute/design).\n- Role of RC1 and RC2 — functionality must be checked into master before RC1 builds are started. Post RC1 it is mostly about corner cases, integration and upgrade tests. Only those features that can complete the upgrade testing within the RC2 timelines are considered for the current release.\n- Role of release manager — Conducts follow-ups via daily standups on pending items and mitigating the risks by seeking additional help or by pushing the feature out of the release.\n- The final two weeks — As we reach the end of a one month release cycle the focus turns to refactoring and adding more test cases while stabilizing the features rather than introducing new features. The last two weeks are also about polishing documentation and trying to reach out to users whose requests have been incorporated into the product to get some early feedback.\n- What else? I haven’t spoken about the role of beta tests or dogfooding of the releases by using OpenEBS in our own hosted services such as OpenEBS director. Perhaps I’ll dig into these in a future blog. Bookkeeping tasks that start after the release also take a lot of time. For example, OpenEBS can be deployed via different partner platforms, each of which maintains their repositories for their Helm charts. Each of these partners are evolving with new guidelines for check-ins and they tend to go at their own pace. There is definitely room for improvement here and hopefully, the way the kubernetes apps are delivered will be standardized so that such bookkeeping tasks can be reduced.\n\n*How do you run your Open Source projects? What tools do you use to improve productivity? Please drop in a comment. Would love to hear from you and improve the care and feeding of the OpenEBS community.*\n\nGetting back to OpenEBS 1.1. The major features, enhancements and bug fixes in this release include:\n\n- Upgrades! Support for the upgrade of OpenEBS storage pools and volumes through Kubernetes Job. As a user, you no longer have to download scripts to upgrade. The procedure to upgrade via Kubernetes Job is provided [here](https://github.com/openebs/openebs/tree/master/k8s/upgrades/1.0.0-1.1.0). Kubernetes Job-based upgrade is a step towards completely automating the upgrades in the upcoming releases. Would love to hear your feedback on the [proposed design](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/upgrade). Note: Upgrade job makes use of a new container image called quay.io/openebs/m-upgrade:1.1.0.\n- CSI — The CSI driver reached Alpha with initial functionality for provisioning and de-provisioning of cStor volumes. Once you have OpenEBS 1.1 installed, take the CSI driver for a spin on your development clusters using the instructions provided [here](https://github.com/openebs/csi). The addition of the CSI driver also requires a shift in the paradigm of how the configuration of the storage class parameters should be passed on to the drivers. We want to keep this seamless, please let us know if you have any inputs on what you notice as some of the nice to have as we shift towards the [CSI driver](https://github.com/openebs/openebs/tree/master/contribute/design/1.x/csi).\n- Day 2 automation ongoing — There is a tremendous amount of work ongoing to further automate Day 2 operations of the cStor storage engine. Most of these changes did not make the current release because the nature of schema changes were larger than could be taken within the current release cycle. The feature is under active development and if you are interested in providing feedback on how this feature is shaping up, you can find the proposed design [here](https://github.com/openebs/openebs/pull/2595). Thank you to everyone that has already chipped in with ideas and feedback.\n\nPerhaps the greatest highlight of this release is an increased involvement from OpenEBS user community pitching in with GitHub Issues as well as providing contributions.\n\n![Giacomo Longo Message](https://cdn-images-1.medium.com/max/800/1*hZ7FK18EK2_PfjdCJB2OTQ.png)\n\nHere are some issues that were raised and fixed within the current release.\n\n- Fixed an issue where backup and restore of cStor volume using OpenEBS velero-plugin was failing when OpenEBS was installed through Helm. [@gridworkz](https://github.com/gridworkz)\n- Fixed an issue with NDM where the kubernetes.io/hostname for Block Devices on AWS Instances was being set as the nodeName. This was resulting in cStor Pools not being scheduled to the node as there was a mismatch between hostname and nodename in AWS instances. [@obeyler](https://github.com/obeyler)\n- Fixed an issue where NDM was seen to crash intermittently on nodes where NVMe devices are attached. There was an issue in the handling of NVMe devices with write cache supported resulting in a segfault. [Private User]\n- Added support to disable the generation of default storage configuration like StorageClasses, in case the administrators would like to run a customized OpenEBS configuration. [@nike38rus](https://github.com/nike38rus)\n- Fixed an issue where the cStor Target would fail to start when the NDM sparse path is customized. [@obeyler](https://github.com/obeyler)\n- Fixed a regression that was introduced into the cStor Sparse Pool that would cause the entire Volume Replica to be recreated upon the restart of a cStor Sparse Pool. The fix was to make sure the data is rebuilt from the peer Sparse pools instead of recreating. Test cases have been added to the e2e pipeline to catch this behavior with Sparse Pools. Note that this doesn’t impact the cStor Pools created on Block Devices. [@vishnuitta](https://github.com/vishnuitta)\n- For Jiva Volumes, created a utility that can clear the internal snapshots created during replica restart and rebuild. For long-running volumes that have gone through multiple restarts, the number of internal snapshots can hit the maximum supported value of 255, after which the Replica will fail to start. The utility to check and clear the snapshots is available [here](https://github.com/openebs/openebs/tree/master/k8s/jiva). [@rgembalik](https://github.com/rgembalik)[@amarshaw](https://github.com/amarshaw)\n- Enhanced velero-plugin to allow users to specify a backupPathPrefix for storing the volume snapshots in a custom location. This allows users to save/backup configuration and volume snapshot data under the same location rather than saving the configuration and data in different locations. [@amarshaw](https://github.com/amarshaw)\n\n***For detailed change summary, steps to upgrade from a previous version, or to get started with v1.1 please refer to:*** [***Release 1.1 Change Summary***](https://github.com/openebs/openebs/releases/tag/1.1.0)\n\nIn short, OpenEBS 1.1 shows that OpenEBS development is marching ahead faster and faster and delivering more and more features, fixes and platforms.\n\nAs always if you have any feedback or inputs regarding the OpenEBS project or project management — please reach out to me on [Slack](https://slack.openebs.io) or [GitHub](https://github.com/openebs/openebs/) or via comments here.\n\n\n\n Example yaml file:\n\n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: default\n type: hostdir\n spec:\n path: \"/mnt/disks/ssd0\"\n ---","notHasFeatureImage":false,"slug":"openebs-community-releases-v1.1,-maintaining-a-faster-release-cadence."},{"id":45,"title":"OpenEBS Project Update and whats coming in v1.0","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"16-05-2019","tags":["Cncf","Containerattachedstorage","Storage","Kubernetes","OpenEBS"],"excerpt":"My heartfelt gratitude to hundreds of contributors and thousands of users and friends of OpenEBS who have contributed to OpenEBS becoming a CNCF Sandbox Project.","content":"\nMy heartfelt gratitude to hundreds of contributors and thousands of users and friends of OpenEBS who have contributed to OpenEBS becoming a CNCF Sandbox Project.\n\nFor those of you who have just heard of OpenEBS or are curious about the current state of OpenEBS, I put together this quick blog that *Looks back to where we started and how far we have come*. (I am pretty sure the GOT fans will get this reference). I then talk about the current release 0.9 and what is to come in 1.0.\n\nThe OpenEBS vision has been pretty clear since the start: provide an Open Source Storage Solution that enables Enterprises and Solution Architects to reap the Agility benefits promised by Container Native Architectures. We decided a few years ago that delivering data agility could best be done by using the tools that deliver agility in development and operations already — cloud native architectures, including containers and Kubernetes. Specifically we decided to move Stateful Workloads and the logic that delivers storage services to protect and manage them into a microservices based architecture deployed via containers. We also set out to enable agility by fighting the lock-in of data into vendor and cloud specific storage silos that lock users into specialized systems and services that themselves require special skills to run and scale.\n\nOver the last couple of years the notion of containerizing stateful workloads and managing them via Kubernetes has become increasingly accepted. We now see a great breadth of deployments of different workloads on different flavors of Kubernetes — you can read more about how the pattern of cloud native or what we call Container Attached Storage is being adopted on the CNCF blog we helped author here: [A Year Later — Updating Container Attached Storage](https://www.cncf.io/blog/2019/05/16/a-year-later-updating-container-attached-storage/).\n\n*Quick point that you may have already gathered: OpenEBS is completely Kubernetes native and if you know how to manage your applications in Kubernetes, you already know how to use OpenEBS.*\n\nSo, OpenEBS started out to build a Storage Solution that has:\n\n- *Stable Date Engines*, can be run on any underlying Kubernetes Platform. And it has got to be multiple storage engines, so administrators can compose or pick the right one for the right workload. For instance, an OpenEBS PV for Mongo can be using a completely different data engine compared to a PV for PostgreSQL or Jenkins.\n- *Standard Specs* to manage the Data Engines, so that administrators can use the existing Kubernetes tooling infrastructure to manage storage as well. Kubernetes Custom Resources and operators are used to manage everything with regards to OpenEBS.\n- *Security* controls that enable running on platforms like OpenShift, SuSE CAAS or in Kubernetes Clusters with strict Pod Security Policy.\n- *Scalable* architecture that can deliver the required performance to the Application, without adding much of an overhead over the underlying storage medium.\n\nAnd above all, it has to be ***simple and easy to use.***\n\nIt feels great for OpenEBS to be approaching 1.0!! We are extremely humbled at the amount of love in terms of both support and scrutiny that we have received over the last couple of years as we went through with building each block by block completely in the Open Source. OpenEBS has clearly established itself as the most simple to use and cost effective storage solution that is available out there to use with Kubernetes — and yet we know there is much more to do, especially as there seems to be a risk of higher level commands such as those covering data mobility being pulled back from Kubernetes into their own systems by proprietary vendors. More on that in a later blog and/or set of GitHub issues :) .\n\nIn the following sections, I go through in a bit more detail the current state of the main aspects of OpenEBS — Stability, Standard Spec (aka Storage Policies), Security Controls and Scalable architecture; followed by a quick summary of the current release (0.9) and what we are working on currently for v1.0.\n\n**Data Engines:** The data engines are the containers responsible for interfacing with the underlying storage devices or cloud volumes such as host filesystem, rotational drives, SSDs and NVMe devices. The data engines provide volumes with required capabilities like high availability, snapshots, clones, etc. Depending on the capabilities required, users can select the right data engine like cStor ( a CoW based) or Jiva or even Local PVs for a given volume.\n\nWe define a data engine as stable if it meets the following criteria:\n\n- Resilient against node, network or storage device errors. No data loss!\n- Ease of management ( including Day 2 Operations).\n- Users are running the data engine in production for more than 6 months and have gone through two or more OpenEBS version upgrades.\n\nThe current state of the 3 data engines supported by OpenEBS are as follows:\n\n- **Jiva (stable)** — The first and basic data engine that was supported by OpenEBS and has been deployed the longest in production by users. We have fixed several issues that came up with regards to cluster upgrades, node migrations, storage expansion and so forth. Ideal for cases where only replication of data is required. For backup and restore, Velero/Restic is used. Very easy to use, lightweight.\n- **cStor (beta)** — The most feature rich data engine that has the support for extremely efficient snapshots and clones. Highly recommended for cases where the nodes have storage devices attached. The current release contains Velero OpenEBS plugins that help with optimizing the backup/restore process. While already used in production by some of our users, we consider this as beta — until all day 2 operations are easily performed.\n- **OpenEBS Local PV (alpha)** — An extension to Kubernetes Local PV, with the plan to ease the management of disks by using the OpenEBS Node Storage Device Manager (NDM). The current OpenEBS 0.9 release contains the initial version of the OpenEBS Local PV Dynamic Provisioner. While we have tested OpenEBS Local PV extensively however per the criteria above it remains alpha as the newest engine. From an IO engine perspective, it is as stable as it can get. :-)\n\nAdditional details and how each of the Data engines operate are provided in this [Presentation](https://docs.google.com/presentation/d/1mjOkAQppyd23sw7PIryxu5kSrex352bT6bINzw6mUFY/edit?usp=sharing)\n\n![OpenEBS cStor Volume](/images/blog/cas-example-openebs-cstor-volume.png)\n\n**Standard Specifications or API:** Standardization is achieved by architecting OpenEBS as a set of microservices using Kubernetes Custom Resources and Operator patterns. The same set of tooling used to manage the applications — like Helm, Prometheus, and Grafana — can also be used to manage OpenEBS itself. The configuration of OpenEBS is completely controlled via YAMLs (Custom Resources) and volumes are provisioned via the Kubernetes Dynamic Provisioners, Storage Classes and Persistent Volume Claims.\n\nThe components of OpenEBS that act on the user requests (via CRs) and generate / launch the Kubernetes Objects — like Deployments, Services and Persistent Volumes are collectively called as the *Storage Management or Control Plane*. The interactions to the Storage Management Plane can achieved via `kubectl`.\n\nThe administrators can customize the OpenEBS parameters by modifying the YAMLs. Some of the configuration that are possible are:\n\n- Setting up Taints and Tolerations to decide the nodes where the Storage Pods need to be deployed.\n- Setting up the Node Selectors or Anti-Affinity parameters to select the correct nodes for launching the Volume (Data Engine) pods.\n\nFor further details check: [https://openebs.io](/docs?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n\nAnother cool feature that has become a favorite of advanced OpenEBS users is that, as Kubernetes introduces new features that have to be passed on to the Dynamically generated Volume (Data Engine) Pods, users can directly patch the pods and also update the Volume Template YAMLs at run time. To learn more about this feature, hit us up on slack: [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294)\n\nIn this regard, as we move towards 1.0, we would like to improve the documentation and governance around maintaining multiple versions of the API.\n\n**Security Controls:** Storage is a cluster add-on service. It needs to access the storage devices — either as hostpath or as block devices that are attached to the nodes. Since OpenEBS is completely Kubernetes native, access rights and privileges required by the individual components are completely transparent and can be controlled by the RBAC configuration by Kubernetes Cluster Administrators. One of the interesting feedback we received as Enterprises started to adopt OpenEBS was that — the IT Teams want to retain the control on the storage and shield the specifics of the node details or scheduling details from application developers.\n\nOpenEBS can now be configured easily to run in:\n\n- Security Enhanced Linux Platforms (selinux=on) like RHEL, CentOS or OpenShift. We talk in more detail about this [here](/blog/configuring-openebs-to-run-with-securityenhanced-linux?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n- OpenEBS can be used on clusters where the default setting is to not grant access to the hostpaths for Developer namespaces. To support this use case, we now support an option to run the Jiva Volumes Pods (that require access to hostpath) to be deployed in OpenEBS Namespace using a *StoragePolicy**—**DeployInOpenEBSNamespace*. The StoragePolicies in OpenEBS are configured via StorageClasses.\n- OpenEBS can be used on clusters enabled with Pod Security Policies. The PSP for OpenEBS has been contributed by a user and can be found [here](https://github.com/openebs/openebs-docs/issues/484).\n\n**Scale and Performance:** OpenEBS is architected to horizontally scalable with nodes — Persistent Volumes spreading out pretty evenly across the nodes or a subset of nodes designated for Storage. However the data of any given Persistent Volume is always fixed to a specific set of nodes, eliminating the need for expensive metadata lookups to find the data blocks as the number nodes get higher. What I like even better is, that the replica tells the controller that it has the data effectively inversing the responsibility from volume target/controller having to know where data is located to replicas telling I have the data.\n\nAnother aspect of OpenEBS Volumes when it comes to performance is that each Volume is completely isolated and doesn’t get impacted by work/load on other Volumes. For example, a node rebuild will not degrade ALL volumes in the cluster.\n\nAs OpenEBS is completely developed in user space and run as Kubernetes Pods, administrators get complete control on the resources like CPU/RAM that should be allocated to Storage. There won’t be cases of kernels hogging all the resources. Administrators also can tune for example the number of threads allocated and parallel IOs supported per Volume — tuning will have an impact depending on the type of workloads (Sequential / Random).\n\nOf course, then we have workloads that require low latency, and need to work to be deployed on nodes with limited storage available, for example a couple of NVMe devices. OpenEBS Local PVs provide the functionality to make use of the Local Storage and help with dynamic provisioning of Local PVs. OpenEBS Local PVs offer an excellent choice for cases like NuoDB where replication is inherently taken by NuoDB itself and the expectation is only to get a persistent storage with node affinity configured for the storage pods. A default storage class called — `openebs-hostpath` is available in the current release. Check it out and let us know what you think.\n\nWe are seeing OpenEBS users progress from running CI/CD workloads in staging to now running critical databases in production on OpenEBS Volumes. If you are interested in performance, we have a OpenEBS user currently exploring and sharing the benchmarking numbers on several different platforms. Join the discussion on our slack — [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294).\n\n— -\n\nWhile I covered earlier some of the items introduced recently in [0.9](https://github.com/openebs/openebs/releases), here is a summary of some significant changes:\n\n- Introduction of Dynamically provisioned OpenEBS Local PVs for making use of the storage available on the nodes itself for running NewSQL kind workloads. Refer to this [blog](/blog/dynamic-provisioning-of-kubernetes-local-pvs-using-openebs?__hstc=216392137.f0da50a3ce0cf28f99a3c60d1f6006cf.1580117750726.1580117750726.1580117750726.1&__hssc=216392137.1.1580117750727&__hsfp=3765904294) to get started with OpenEBS Local PVs.\n- Enhanced the cStor Replica distribution logic for MongoDB or Cassandra Statefulsets to provide storage high availability and reducing the performance overhead\n- Backup and Restore processes for cStor Volumes using [OpenEBS Velero Plugin](https://github.com/openebs/velero-plugin) that can perform incremental snapshot backup and restore.\n- Enhance the Deployment and Placement of the Jiva Volume Pods to facilitate working in environments with strict Pod Security Policies and for Kubernetes clusters that tend to see a lot of pod evictions or node drains.\n- Introduced Web Admission Hook that will help with validation and avoiding misconfigurations.\n- Developed an upgrade framework using the CAS Templates that will help developing operator based upgrades from earlier releases to the current release. We are excited to use this framework for upgrading from 0.8.2 to 0.9\n- Enhanced the Prometheus exporters to support generating cStor Volume Replica metrics.\n- As always many other user reported issues made into this release. To learn more checkout the [release notes](https://github.com/openebs/openebs/releases).\n\nEach OpenEBS release goes through Litmus — GitLab Pipelines that verify new functionality, backward compatibility and also acceptance from users through pre-release build testing. Please reach out to us if you would like to be included in the pre-release notifications.\n\nI am very excited about the following active contributions that are filling in the gaps for 1.0\n\n- BDD tests that developers can execute as part of the feature development. This is an extension to the Sanity tests are executed in Travis CI and Litmus Tests in GitLab CI.\n- Updating the Design, Contributor and Governance related details with the help of CNCF guidelines.\n- Freezing on the cStor Specs based on the feedback received from users on how to make it more user friendly and easy to manage. The NDM Disk Specification has also changed to BlockDevice spec as per the feedback received in making it generic to Storage Devices.\n- Additional enhancements automatic the cStor Day 2 operations with regards to scaling up the capacity of cStor Pool by increasing the size of the underlying disks, scaling up and down the number of cStor Pools in a given cluster.\n- Support for integrating the OpenEBS Local PV into the BlockDevices discovered and managed by NDM.\n- Support for ARM builds.\n\nFor contributing to the above feature or learning more about them, you can reach out to us on the #contributors channel or check out the [milestones](https://github.com/openebs/openebs/milestones).\n\nI am very excited to be at KubeCon Europe and to meet in person some of the hundreds of contributors and many thousands of users of OpenEBS who have all made it possible for OpenEBS to be recently accepted as a CNCF Sandbox Project. Thank you Again!\n\nMayaData will be speaking about OpenEBS at the KubeCon events in Barcelona in the Cloud Native Storage Day and Open Data Autonomy Mini Summit on May 20th, and also at booth SE41 until Thursday in KubeCon Expo Hall.\n","notHasFeatureImage":false,"slug":"openebs-project-update-and-whats-coming-in-v1.0"},{"id":46,"title":"Configuring OpenEBS to run with Security-Enhanced Linux","author":"Akhil Mohan","author_info":"Software Engineer @ MayaData, working on Cloud Native Tech.","date":"10-05-2019","tags":["Centos","Kubernetes","OpenEBS","Selinux","Uncategorized","Openshift","Tutorials"],"excerpt":"In this short How-To blog, I will walk you through the steps for running OpenEBS with Security Enhanced Linux (SELinux=on).","content":"\nIn this short How-To blog, I will walk you through the steps for running OpenEBS with Security Enhanced Linux (SELinux=on). This applies to both OpenShift based Kubernetes Cluster or Clusters using CentOS.\n\nOpenEBS can be treated as a Cluster Add-on service, that can convert the existing Kubernetes Cluster into a full fledged Storage Controllers — that can serve Persistent Volumes to Stateful Workloads while making use of the storage attached to the nodes. Since OpenEBS runs in containers and has to access the underlying storage devices, it needs to run in Privileged mode.\n\nThe component of OpenEBS ( Node Device Manager or Node Disk Manager or simply referred as NDM), is responsible for discovery of the devices and filtering out devices that should not be used by OpenEBS; for example the disk that has OS filesystem. Earlier, to detect the OS disk, the NDM pod by default mounted the `/proc/1/mounts` file, which is restricted on nodes that have SELinux=on. This is now fixed by mounting the `/proc` directory of the host inside the container and then loading the `mounts` file.\n\nSo at a high level, to allow OpenEBS to run in privileged mode in SELinux=on nodes, the cluster should be configured to grant privileged access to OpenEBS service account.\n\nHere are the steps I have followed:\n\n**Step 1: Setup appropriate security context for OpenEBS**\n\n**On OpenShift Clusters:** Select the right SCC for OpenEBS\n\nOpenEBS like other cluster add-on services requires its pods to be executed with privileged access as it needs to directly access the block devices. However, this will not mean that SELinux should be turned off.\n\nWhile running on OpenShift, we need to be aware of the security constraints available and to select the right security constraint for OpenEBS. There are 7 SCCs available.\n\n![Security Context Constraints (SCC) in OpenShift](/images/blog/security-context-constraints-(SCC)-OpenShift.png)\n\nTo have OpenEBS pods running in privileged mode, add the OpenEBS service account (openebs-maya-operator) to use the privileged SCC; OpenShift will take care of setting the correct permissions (or SELinux labels) to make the container run in privileged mode. Sample commands to do this:\n\n oc adm policy add-scc-to-user privileged system:serviceaccount:openebs:openebs-maya-operator\n\n**On CentOS/RHEL:** Enable Pod Security Policies for OpenEBS\n\nCreate a file `openebs-privileged-psp.yaml` with the below spec.\n\n apiVersion: extensions/v1beta1\n kind: PodSecurityPolicy\n metadata:\n name: openebs-privileged\n annotations:\n seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*'\n spec:\n privileged: true\n allowPrivilegeEscalation: true\n allowedCapabilities:\n - '*'\n volumes:\n - '*'\n hostNetwork: true\n hostPorts:\n - min: 0\n max: 65535\n hostIPC: true\n hostPID: true\n runAsUser:\n rule: 'RunAsAny'\n seLinux:\n rule: 'RunAsAny'\n supplementalGroups:\n rule: 'RunAsAny'\n fsGroup:\n rule: 'RunAsAny'\n\nThen apply the YAML file\n\n kubectl apply -f openebs-privileged-psp.yaml\n\n**Step 2: Install OpenEBS**\n\nDownload the latest version of `openebs-operator.yaml` file.\n\n wget https://openebs.github.io/charts/openebs-operator-0.8.2.yaml\n\n**On CentOS/RHEL:** An extra step of adding PSP to the Cluster Role is required.\n\nEdit the ClusterRole in the YAML to add `openebs-privileged` PSP\n\n # Define Role that allows operations on K8s pods/deployments\n kind: ClusterRole\n apiVersion: rbac.authorization.k8s.io/v1beta1\n metadata:\n name: openebs-maya-operator\n rules:\n - apiGroups: [\"*\"]\n resources: [\"nodes\", \"nodes/proxy\"]\n verbs: [\"*\"]\n - apiGroups: [\"*\"]\n resources: [\"namespaces\", \"services\", \"pods\", \"deployments\", \"events\", \"endpoints\", \"configmaps\", \"jobs\"]\n verbs: [\"*\"]\n - apiGroups: [\"*\"]\n resources: [\"storageclasses\", \"persistentvolumeclaims\", \"persistentvolumes\"]\n verbs: [\"*\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshots\", \"volumesnapshotdatas\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"apiextensions.k8s.io\"]\n resources: [\"customresourcedefinitions\"]\n verbs: [ \"get\", \"list\", \"create\", \"update\", \"delete\"]\n - apiGroups: [\"*\"]\n resources: [ \"disks\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"storagepoolclaims\", \"storagepools\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"castemplates\", \"runtasks\"]\n verbs: [\"*\" ]\n - apiGroups: [\"*\"]\n resources: [ \"cstorpools\", \"cstorvolumereplicas\", \"cstorvolumes\"]\n verbs: [\"*\" ]\n - apiGroups: ['extensions']\n resources: ['podsecuritypolicies']\n verbs: ['use']\n resourceNames:\n - openebs-privileged\n - nonResourceURLs: [\"/metrics\"]\n verbs: [\"get\"]\n\nInstall OpenEBS\n\n kubectl apply -f openebs-operator-0.8.2.yaml\n\n**Note: If you are using helm to install openebs, you will need to apply the above change after it has been installed. In a future release of the helm chart, I will work on making this configurable parameter.**\n\n**Step 3: (Optional) Create a new cStor Pool.**\n\nYou can skip this step if using the default cStor Sparse pool.\n\n**Step 3a**: Verify all pods are working and cStor Pools are running\n\n![List of all pods in openebs namespace after installation](/images/blog/pod-lists.png)\n\n**Step 3b**: Verify that disks available on the nodes are discovered.\n\n kubectl get disks\n\n\n![Disks detected by NDM, along with sparse disks](/images/blog/ndm-detected-disks.png)\n\n**Step 3c**: Create a storage pool claim using the instructions at [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools)\n\nCreate a `cstor-pool-config.yaml` as mentioned in the docs.\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-pool1\n annotations:\n cas.openebs.io/config: |\n - name: PoolResourceRequests\n value: |-\n memory: 1Gi\n - name: PoolResourceLimits\n value: |-\n memory: 2Gi\n spec:\n name: cstor-pool1\n type: disk\n maxPools: 3\n poolSpec:\n poolType: striped\n disks:\n diskList:\n - disk-301722dba60542a05ec2817773ff11ed\n - disk-a2d99f96090f0675b7ea60925b58414d\n - disk-cd01d33213c3a42c6b1a9f0798277368\n\nApply this file `kubectl apply -f cstor-pool-config.yaml`\n\n![3 cStor pool pods will be running](/images/blog/cstor-pool.png)\n\n**Step 3d**: Create a new storage class using SPC as `cstor-pool1` or edit the default storage class to use the newly created SPC. I have edited the already available default storage class.\n\n**Step 4: Running Percona Application**\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-cstor-sparse-deployment.yaml\n\nEdit the file to remove security context from the percona deployment.\n\nApply the yaml file\n\n kubectl apply -f percona-openebs-cstor-sparse-deployment.yaml\n\n\n![The percona cstor pod up and running in default namespace](/images/blog/percona-pod.png)\n\nThe `percona-cstor` pod will be up and running.\n\nThanks for reading my blog! We continue to look for feedback and suggestions on how we can improve OpenEBS integration with all possible platforms and workloads. If you have any feedback on my blog or see any area of improvement with OpenEBS in general, please provide feedback below or find us on Twitter or on the OpenEBS slack community [http://slack.openebs.io](http://slack.openebs.io/).\n","notHasFeatureImage":false,"slug":"configuring-openebs-to-run-with-security-enhanced-linux"},{"id":47,"title":"How to Easily Build a CI Dashboard","author":"Chandan Kumar","author_info":"Software Engineer at MayaData Inc","date":"26-04-2019","tags":["CI Dashboard","Git","Kubernetes","Pipeline","GitLab"],"excerpt":"In this tutorial, we’ll go through all the necessary steps for setting up the CI dashboard","content":"\nA code that is never executed for users is essentially a digital waste product. To prevent this building of waste and to showcase the results of code on the Kubernetes environment, we can use the CI Dashboard, along with chaos testing.\n\nThe CI dashboard allows users to view the commit and release-based build and run a chaos test using litmus on a different platform with different versions of Kubernetes.\n\nIn this tutorial, we’ll go through all the necessary steps for setting up the CI dashboard:\n\n1. *Create a project*\n2. *Push the code to GitHub*\n3. *Setup the CI of the project using GitLab*\n4. *Select the chaos test from [*Litmus*](https://github.com/openebs/litmus)*\n5. *Add the script in GitLab YAML to create pipelines for executing the chaos tests*\n6. *Build a CI dashboard(ex: *[*openebs.ci*](https://openebs.ci/)* ) to display the gitlab pipeline history and status*\n7. *Conclusion*\n\n### Step 1: Create a project\n\nFirst, create a [project](https://github.com/openebs/maya) and write some automated testing for it. You should also add the Dockerfile in the project to set up the CI.\n\n### Step 2: Put the codes on GitHub\n\nCreate a repository on GitHub and add a `.gitignore` file to ignore the auto-generated folder or file. Follow the script below to put changes into GitHub.\n\n $ git init\n $ git add .\n $ git commit -s -m \"Initial commit\"\n $ git remote add origin .git\n $ git push origin master\n\n### Step 3: Setup the CI using gitlab\n\nAdd a *.gitlab.yaml* file to project and write the build and test steps. \n(Ex: [https://github.com/openebs/maya/blob/master/.gitlab-ci.yml](https://github.com/openebs/maya/blob/master/.gitlab-ci.yml)). Import the project in gitlab from GitHub. Setup the gitlab pipeline environment variable to push the docker image, or any other, if required. Add the pipeline trigger command in *.gitlab.yaml* file.\n\n### Step 4: Selection of chaos test\n\nSelect the chaos test (litmus book) from the [litmus](https://github.com/openebs/litmus) repository or write your own litmus book if needed. This will be used to test the product performance on different Kubernetes versions and with different cloud vendors.\n\n### Step 5: Add script in gitlab.yaml\n\nCreate a [repository](https://github.com/openebs/e2e-packet) for the execution of the platform-based pipeline. Add `.gitlab.yaml` file and related script to create a cluster, or use the executing cluster and run the different chaos tests in various stages of the [pipeline](https://gitlab.openebs.ci/openebs/e2e-packet/pipelines).\n\nReference the `.gitlab.yaml` file\n[https://raw.githubusercontent.com/openebs/e2e-packet/master/.gitlab-ci.yml](https://raw.githubusercontent.com/openebs/e2e-packet/master/.gitlab-ci.yml)\n\n cleanup-packet:\n when: always\n image: chandankumar4/packet:v4\n dependencies:\n - packet-cluster\n stage: CLUSTER-CLEANUP\n script: \n - chmod 755 ./script/packet-cleanup\n - ./script/packet-cleanup\n \n\n### Step 6: Build a CI dashboard\n\n![CI Dashboard](https://lh4.googleusercontent.com/hoDf2G6VnpIhhkmkQXlF07ocFRm7bJjP5f1ZkA8TZCT6PXMOPkdCO966EecYpk7koCbHPKdMemOA3_kYz8M5qrvLevRDJPw2c0MfYn-yp-iLn4j-qV8wpwT_av2iBYBuMH-4EUeB)\n\nCreate a project called [Ci dashboard backend](https://github.com/openebs/ci-e2e-dashboard-go-backend) that will fetch the pipeline details from gitlab by accessing their API and exposing the same on different API after some enhancement. Create another project called [Ci dashboard](http://github.com/openebs/ci-e2e-dashboard) that will display the gitlab pipeline details by accessing the data from the back end API.\n\n### Step 7: Conclusion\n\nCI dashboard will display the build history of the imported project and analyze the performance on different platforms and versions of Kubernetes.\n\n### References\n\n[https://openebs.ci/](https://openebs.ci/)\n\n[openebs/ci-e2e-dashboard](https://github.com/openebs/ci-e2e-dashboard)\n[Contribute to openebs/ci-e2e-dashboard development by creating an account on GitHub.github.com](https://github.com/openebs/ci-e2e-dashboard)\n\n[openebs/ci-e2e-dashboard-go-backend](https://github.com/openebs/ci-e2e-dashboard-go-backend)\n[OpenEBS CI Dashboard backend using Go and PostgreSQL. — openebs/ci-e2e-dashboard-go-backendgithub.com](https://github.com/openebs/ci-e2e-dashboard-go-backend)[openebs/maya](https://github.com/openebs/maya)\n[OpenEBS Maya extends Kubernetes capabilities to orchestrate CAS containers. — openebs/mayagithub.com](https://github.com/openebs/maya)\n","notHasFeatureImage":false,"slug":"how-to-easily-build-a-ci-dashboard"},{"id":48,"title":"Deploying OpenEBS on SUSE CaaS platform","author":"Ashok Babu","author_info":"Senior DevOps consultant at Wipro works on App Anywhere & cloud-native technologies.","date":"24-04-2019","tags":["Kubernetes","Cloud Native Storage","MayaData","SUSE","Container attached storage","OpenEBS"],"excerpt":"I am recently introduced to OpenEBS, an easy to use persistent storage option for Kubernetes and found it nicely working on the SuSE CaaS platform where I could certify the cloud native databases. ","content":"\nI am recently introduced to OpenEBS, an easy to use persistent storage option for Kubernetes and found it nicely working on the SuSE CaaS platform where I could certify the cloud native databases. In this blog, I cover a few quirks to get it running on the SuSE CaaS platform.\n\nSuSE CaaS Platform is an enterprise-class container management solution that leverage Kubernetes as the orchestration layer and SuSE MicroOS as the host operating system for master and worker nodes.\n\nSuSE CaaS provides enhanced security policies such as predefined pod security policies.\n\nIn SuSE MicroOS, a read-only Btrfs file system is used for the root file system with OverlayFS. Sub-volumes for data sharing is read-write.\n\nOutlined below are some of the steps that need to be taken care while installing OpenEBS version 0.8.1 on SuSE CaaS Platform 3\n\n## Issue\n\nDefault OpenEBS installation would fail on SuSE CaaS platform due to the following restrictions by the platform\n\n- NDM Daemonset fails to spin up as it requires privilege permission\n- Sparse pools will not be created as it uses by default /var/openebs directory which is read-only directory under root filesystem in SuSE CaaS.\n- Runtasks in operator uses /var/openebs directory for temporary file creation due to this cstor-target pods or cstor-pool pod gets stuck in “ContainerCreating” status\n\n## Resolution\n\n### Step1\n\nTo install OpenEBS on SuSE platform run the following custom YAML file instead of default operator YAML\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-susecaas.yaml\n\nThis is a customized operator YAML file for SuSE CaaS platform that uses the role `suse:caasp:psp:privileged` for creating privileged DaemonSet\n\n[https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#ex.admin.security.pod_policies.daemonset](https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#ex.admin.security.pod_policies.daemonset)\n\n### Step2:\n\nPerform the following changes in the runtask after completing step1\n\nNote: — This step is not required if you are using the OpenEBS version 0.9 which is the upcoming release.\n\n- **Change the path in runtask for cstor-pool-create-putcstorpooldeployment**\n\n kubectl edit runtask cstor-pool-create-putcstorpooldeployment-default-0.8.1 -n openebs\n\n Change from\n\n path: /var/openebs/shared-\n\n Change to\n\n path /var/lib/overlay/openebs/shared-\n\n- **Change the path in runtask for cstor-volume-create-puttargetdeployment**\n\n kubectl edit runtask cstor-volume-create-puttargetdeployment-default-0.8.1 -n openebs\n\n Change from\n\n path: /var/openebs/shared--target\n\n Change to\n\n path: path /var/lib/overlay/openebs/shared--target\n\n### Step 3:\n\n**Optional:** If you need to use sparse pool\n\n kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/5860c0a4619a9feddf5d75d11f50f2ea8fdcec82/k8s/demo/fio/demo-cstor-sparse-pool-limits.yaml\n\n### Step 4:\n\nConfiguration of storage pool, storage class and PVC are like any other platform and the steps are outlined in [https://openebs.io/docs](/docs?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294)\n\nPool Configuration — [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#manual-mode)\n\nStorage class — [https://openebs.io/docs/deprecated/spc-based-cstor#creating-cStor-storage-class](/docs/deprecated/spc-based-cstor#creating-cStor-storage-class?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#creating-a-new-class)\n\nVolume — [https://openebs.io/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume](/docs/deprecated/spc-based-cstor#provisioning-a-cStor-volume?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294#provision-from-a-disk-pool)\n\n## Conclusion:\n\nAbove approach can be followed to install OpenEBS 0.8 on SuSE CaaS platform 3.0 which needs additional configuration.\n\nWith OpenEBS 0.9 it would be simplified such that you would need to only apply the operator YAML to perform the installation.\n\n**SUSE CaaS Platform:** (Container as a Service Platform) is an integrated software platform which automates the process of building, managing and upgrading of Kubernetes clusters. It combines the benefits of an enterprise-ready operating system with the agility of an orchestration platform for containerized applications. More details — [https://www.suse.com/products/caas-platform/](https://www.suse.com/products/caas-platform/)\n\n**OpenEBS:** OpenEBS is the leading open-source project for container-attached and container-native storage on Kubernetes. OpenEBS adopts Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller. OpenEBS implements granular storage policies and isolation that enable users to optimize storage for each specific workload. OpenEBS runs in userspace and does not have any Linux kernel module dependencies. More details — [https://openebs.io/](https://openebs.io/?__hstc=216392137.a6c0b8ba8416b65c52c0226c0e0b69fd.1579867391229.1579867391229.1579867391229.1&__hssc=216392137.1.1579867391230&__hsfp=3765904294)\n\nThey have a very responsive community. Visit [https://slack.openebs.io](https://slack.openebs.io)\n","notHasFeatureImage":false,"slug":"deploying-openebs-on-suse-caas-platform"},{"id":49,"title":"High Availability For Jira Using OpenEBS","author":"Patrick Hoolboom","author_info":"Works @MayaData","date":"21-04-2019","tags":["OpenEBS","Kubernetes","K8s","Jira","Docker","Mayaonline"],"excerpt":"Up time, high availability, disaster recovery. Anyone that has been working in development or IT operations roles knows these terms. Maintaining these things is always touted as the most critical of critical tasks.","content":"\nUp time, high availability, disaster recovery. Anyone that has been working in development or IT operations roles knows these terms. Maintaining these things is always touted as the most critical of critical tasks. Even with that in mind, we often slack in these areas. We don’t test our backups or run our applications with high availability in mind. One of the main factors that lead to these less than ideal deployments is that implementing a good high availability story, or testing our backups, adds time and/or complexity to our already busy day. All that being said, let’s take a look at a dead simple solution for one of those applications that many of us run: Jira.\n\nThe standard answer to making Jira highly available and provide a good disaster recovery story is to use Jira Data Center and follow this guide:\n\n[Jira Data Center Guide](https://confluence.atlassian.com/enterprise/jira-data-center-472219731.html)\n\nThat guide shows you how to cluster multiple instances of the application behind a load balancer, then use a shared file system and a shared database underneath. This blog is not about the application layer or even the database (though we do have some fantastic guides for deploying databases using OpenEBS in our documentation). This is going to focus on a simple way to create a replicated storage layer specifically for the file system. Jira stores information on disks such as issue attachments, import/export files, and logos. These are all important in keeping Jira working correctly.\n\nDeploying Jira on Kubernetes using OpenEBS is as simple as installing OpenEBS on Kubernetes, define a storage pool, define a storage class, define your persistent volume claim, and deploy the Jira container. That’s it… and if you are already using OpenEBS it is even simpler. Now, as for the specifics of how to do those things, see this guide:\n\n[Jira - OpenEBS docs](/docs/stateful-applications/jira?__hstc=216392137.fb75a0ac1e54cb037dfbafd0edf1ad3f.1579868085240.1579868085240.1579868085240.1&__hssc=216392137.1.1579868085240&__hsfp=3765904294)\n\nBy leveraging the power of replicas within OpenEBS we have quickly added fault tolerance to the storage attached to our Jira container. It was as simple as defining a replica count in the storage class. Since the replicas are spread across the cluster we no longer have to worry about the storage being a single point of failure. If one disk goes down, the controller will automatically route to one of the replicas with no intervention necessary.\n\nAs you can see OpenEBS has greatly simplified the process of making Jira more resilient in a containerized world. The ease of use of container attached storage makes tasks like these much simpler. It allows us to spend more time working on improving our applications or infrastructure, and less time worrying about its resiliency.\n","notHasFeatureImage":false,"slug":"high-availability-for-jira-using-openebs"},{"id":50,"title":"A Primer on OpenEBS Continuous Integration","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"02-04-2019","tags":["Chaos Engineering","Continuous Integration","E2e Kubernetes Litmus"],"excerpt":"The OpenEBS project consists of several components (control plane & data plane) that directly support the dynamic provisioning & management of a persistent storage volume on the Kubernetes cluster.","content":"\n**This article is part of a #Concepts series on Kubernetes and Litmus. Here, we present an overview of the OpenEBS CI infrastructure with its Litmus powered e2e pipelines.**\n\nThe OpenEBS project consists of several components (control plane & data plane) that directly support the dynamic provisioning & management of a persistent storage volume on the Kubernetes cluster. As with any microservice-oriented system following the DevOps paradigm, there is a need to continuously build and test each component, both in isolation (via unit tests) as well as in relation with the other pieces (integration tests) with emphasis on standard end-user scenarios (e2e). Factor in the need for basic interoperability verification (in terms of supported OS/Platform/Cluster versions) and you have the requirements for the CI framework spelt out.\n\nThe OpenEBS CI infrastructure is based on the Cloud-Native GitLab CI framework which is setup to monitor commits to the core components such as [Maya](https://github.com/openebs/maya), [Jiva ](https://github.com/openebs/jiva)& [cStor](https://github.com/openebs/zfs) and also [e2e ](https://github.com/openebs/e2e-packet)(reduces turnaround time to verify test case sanity). It uses Litmus to drive the e2e pipelines, thereby providing a reference implementation for a Litmus-Experiment based e2e suite.\n\n## GitLab Infrastructure\n\nSome of the reasons for adopting GitLab as the CI framework of choice (amongst standard benefits such as tight integration with our existing SCM, 2-factor auth, webhook support, well-defined UI with pipeline graphs, etc was the need to move away from a plugin-based model (Jenkins thrives on plugins, which may not always be advantageous) to a self-contained platform that supports simple pipeline definitions (`.gitlab-ci.yaml` is far easier to maintain than the groovy-based `jenkinsfile`!). GitLab also offers a more mature kubernetes-native solution that gives users the ability to dogfood OpenEBS storage as the back-end store (PostgreSQL) for the GitLab server.\n\n![GitLab server](/images/blog/openebs-control-and-data-plane.png)\n\nThe GitLab server (with its microservices such as Unicorn, Shell, Workhorse, Registry, Sidekiq, Gitaly, PostgreSQL, Redis, Minio) is hosted on a multi-node bare-metal OpenShift cluster and is configured with pull-based repository mirroring of the OpenEBS component GitHub repos and a webhook based setup that triggers the pipelines upon commits.\n\nWhile Maya and Jiva repos are mapped to shell-based executors due to certain build and integration-test requirements, cStor (zfs) and e2e repos are mapped to docker-machine based executors. The docker-machine executors are inherently auto-scaling, a necessary feature for e2e builds as multiple parallel jobs are spawned during the course of e2e pipelines.\n\n## OpenEBS CI Workflow\n\nEach commit to the component source triggers the “gitlab build” procedure, which can be split into two logical phases: “component build” and “e2e.”\nThe component build executes the respective makefile which typically involves running unit tests, building the GO binaries, creating docker images, running integration tests and pushing docker images tagged with the commit SHA to the respective container repositories. It also performs certain pre-e2e routines before finally triggering the e2e pipelines.\n\nThe e2e phase involves running several parallel pipelines (based on Kubernetes cluster platforms or versions), with each pipeline containing multiple stages such as test bed setup, application deployment, Litmus experiments (functional and chaos), and finally clean-up. Needless to say, the component versions used are the ones built in the previous phase.\n\n![OpenEBS CI Workflow](/images/blog/openebs-ci-flow.jpg)\n\n**Note**: *Currently, the GitLab CI works in a “retrospective mode,” as it is invoked on commits to the upstream branches. There is work in progress to extend the support for pull requests (presently, a travis-based build verifies commit sanity to aid PR acceptance).*\n\n## Baseline Commit\n\nBefore triggering the e2e pipelines as part of the final step, the build pipeline performs a pre-e2e routine to generate metadata about the impending e2e run. As is evident from the previous discussion, the e2e pipelines are triggered against commits to any of the OpenEBS component repos (maya, jiva, zfs/cstor, e2e). Images pushed as part of the build pipeline are deployed during the e2e. Therefore, it is necessary to baseline or identify an e2e run against the primary trigger (commit) while maintaining the details of image versions of the other relative components.\n\nThis is achieved by writing the details of the baseline commit (timestamp, component repo, branch and commit ID) into the file head of a “baseline artifact” maintained in a separate repository. Once the e2e pipeline is initiated, the baseline artifact is [parsed ](https://github.com/openebs/e2e-infrastructure/blob/master/env/env_exporter.py) for the “most current” image tags of each component, which will invariably include the current baseline commit and the latest ones for other components, in the test-bed preparation stage. This information is then used to [precondition ](https://github.com/openebs/e2e-infrastructure/blob/master/env-update/env.py) the OpenEBS Operator manifest before its deployment on the test clusters.\n\n## OpenEBS e2e Pipelines: Leveraging Litmus\n\n*One of the arguments against the inclusion of e2e as part of CI pipelines is that they could be flaky and time-consuming (under ideal circumstances it involves testing every moving part of the microservice and needs more maintenance). However, the e2e can confirm achievement of the goals that the solution was conceived and built to accomplish. The extent of coverage and the schedules can be optimized for development agility, but at OpenEBS, we feel it is a must-have in our CI pipelines.*\n\nOpenEBS CI makes use of Litmus to drive its e2e pipelines, right from test bed creation (cluster creation playbooks) all the way through the e2e tests (litmus experiments). The e2e pipeline involves several stages, with one or more gitlab jobs scheduled to run in a given stage. Each gitlab job is associated with a “runner script” that runs an “e2e test.” This in turn invokes/executes a litmus experiment (or litmus ansible playbook in the case of cluster creation/destroy jobs).\n\nThe various stages in the e2e pipeline are discussed below:\n\n![Various stages in the e2e pipeline](/images/blog/e2e-pipelines.png)\n\n**Cluster Creation**: This stage calls up the Kubernetes cluster by executing the platform-specific playbooks. Cluster parameters are controlled via runtime arguments. The artifacts generated upon this job’s execution such as cluster config, which includes kubeconfig and cluster resource names, are passed over to subsequent stages as dependencies. The Litmus pre-requisites are also installed once the cluster is created. Currently, Litmus supports creation of clusters on these platforms:\n\n- Baremetal Cloud: Packet (ARM based physical servers)\n- Managed Kubernetes: GKE, EKS, AKS\n- Cloud Hosted/Self-Installed: AWS, GCP (via KOPS)\n- On-Premise: OpenShift (vSphere VMs)\n\n**Cluster Provision**: Provision equips the cluster with additional disk resources native to the specific platform (GPD, EBS, Packet Block Storage, Azure Block Device) used by the storage-engines as physical storage resources.\n\n**Provider Setup:** Here, the system deploys the customized/preconditioned OpenEBS Operator manifest (based on the baseline commit) on the cluster, thereby setting up the control plane and preparing default storage pool resources. The logging infrastructure (fluentd) is also setup on the created cluster.\n\n**Stateful Application Deployment**: The OpenEBS e2e verifies interoperability with several standard stateful applications such as Percona-MySQL, MongoDB, Cassandra, PostgreSQL, Prometheus, Jenkins, Redis etc. These applications are deployed with OpenEBS storage classes (tuned for each app’s storage requirement). Each application is accompanied by respective load-generator jobs that simulate client operations and real-world workloads.\n\n**App Functionality Tests:** Each deployed application is subjected to specific behavioural tests such as replica scale, upgrade, storage resize, app replica re-deployment, storage affinity etc. Most of these tests are common day-2 operations.\n\n**Storage/Persistent Volume Chaos Tests:** The PV components such as controller/replica pods are subjected to chaos (pod crash/kill, lossy networks, disconnects) using tools such as ChaosKube, Pumba, and Kubernetes APIs (via kubectl) to verify data availability and application liveness during adverse conditions.\n\n**Infrastructure Chaos Tests:** The cluster components such as storage pools, nodes, and disks are subjected to different failures using Kubernetes APIs (forces evicts, cordon, drain) as well as platform/provider specific APIs (gcloud, awscli, packet) to verify data persistence and application liveness.\n\n**Stateful Application Cleanup:** The deployed apps are deleted in this stage, thereby verifying de-provisioning and cleanup functionalities in the OpenEBS control plane.\n\n**Cluster Cleanup:** The cluster resources (nodes, disks, VPCs) are deleted. With this final step, the e2e pipeline ends.\n\n## GitLab e2e Job Runner Template\n\nEach GitLab job running the e2e test executes (bash) scripts containing steps to run and monitor a Litmus experiment. These scripts are invoked using desired arguments, specified as part of the job definition in the e2e repository’s `.gitlab-ci.yml`. The standard template maintained in these (bash) runner scripts and the performed tasks are described below.\n\n![GitLab runner template(E2E)](/images/blog/gitlab-job-runner.png)\n\n**Generate Unique Test Name:** Each gitlab job is associated with a litmus experiment that has a test/experiment name. The result of this litmus experiment is stored in a Litmus Custom Resource (CR) of the same name. The success of a test and therefore the gitlab job is derived from this CR. Occasionally, it is possible that the same litmus experiment is run against different applications or storage engines in a pipeline, thereby necessitating a unique element or ID in the CR name. In this step, a user-defined input (run_id) is accepted to generate a unique test/CR name.\n\n**Setup Dependencies:** Depending on the nature of the gitlab job (cluster create/delete playbooks OR litmus experiments), the executor machine is updated with the appropriate directory structure and target cluster info (such as cluster configuration file, cluster names, disk details, VPC information etc.) to ensure successful test execution.\n\n**Precondition Litmusbook:** Each litmusbook (the Kubernetes job specification YAML) consists of a default set of test inputs such as placeholders for application, storage, chaos info, etc. These are overridden/replaced by desired values in this step. In addition, the default name of the test is replaced with the unique name generated by the runner at the start of execution.\n\n**Run Litmus Experiment:** The litmusbook is deployed and monitored for completion, with a polling interval of 10s. The status of both the litmus Kubernetes job and the ansible-runner container is checked as necessary and sufficient conditions, respectively, to determine completion of the litmus experiment.\n\n**Get Litmus Experiment Result:** The result CR with the unique name generated is queried to determine the Litmus experiment result. The runner script completes execution with a zero/non-zero exit code depending on a pass/failure result, thereby setting the gitlab job status.\n\n**Logging Framework for e2e**\n\nOpenEBS CI uses the popular EFK (Elasticsearch-Fluentd-Kibana) stack as the logging framework for the e2e pipelines. Each target cluster brought up as part of the e2e pipeline is configured with the fluentd-forwarder daemonset and fluentd-aggregator deployment, with the latter streaming the logs to the remote ElasticSearch instance running on the master (GitLab-CI) cluster. These are then rendered by the Kibana visualization platform that is also running on the master cluster. The forwarders are configured with the appropriate filters based on pipeline and commit IDs to aid in a quick data analysis.\n\n![EFK-Based Logging framework](/images/blog/efk-based-logging-framework.png)\n\n**Conclusion**\n\nHopefully, this article has provided you with a better understanding of the CI/E2E practices in the OpenEBS project. CI is an important factor in contributor happiness as well as user confidence, and we are focused on continually making it more robust. Feel free to share your questions, comments, and feedback with us — we are always listening!\n","notHasFeatureImage":true,"slug":"a-primer-on-openebs-continuous-integration"},{"id":51,"title":"Dynamic provisioning of Kubernetes Local PVs using OpenEBS","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-03-2019","tags":["Kubernetes","Open Source","OpenEBS","Stateful Applications","Storage"],"excerpt":"In this blog I discuss why we are adding the support of Local Persistent Volumes to the open source OpenEBS -  I view this as a significant step forward in our mission of delivery true data agility to the broader cloud native community and I welcome your feedback.","content":"\n*Updated May 16th 2019: The alpha version of the OpenEBS Local PV provisioner has been included in OpenEBS Release 0.9. This blog is updated with the setup instructions and examples from v0.9.*\n\nIn this blog I discuss why we are adding the support of Local Persistent Volumes to the open source OpenEBS — I view this as a significant step forward in our mission of delivery true data agility to the broader cloud native community and I welcome your feedback.\n\nThere are many stateful workloads like Mongo, Redis, Cassandra, Postgres, Nuodb etc.,that are capable of performing their own replication for high availability and basic data protection, eliminating the need for the underlying storage to copy or replicate the data for these purposes. Instead, what users of these workloads seem to really want from underlying storage is maximum throughput that can be delivered along with of course the guarantee that data is consistent on the disk — and increasingly we are seeing that the operators of these workloads also want to use Kubernetes as their fundamental API irrespective of what IaaS or hardware or Kubernetes service underlies their environment.\n\nKubernetes provides an option to use Host Paths (and very recently the use of Local PVs is also gaining popularity) which present a very simple and elegant option to get started. The Local PVs differ from the Host Paths in two aspects:\n\n- Application developers have to specify the Host Path in their Pod Spec.\n- Host Paths typically share a single underlying disk, where as Local PVs occupy the entire disk.\n\nThere are many scenarios where Host Paths are the only option since the Kubernetes Nodes can have a limitation on the number of Disks (or devices) that can be attached — either due to node capabilities or just for cost purposes.\n\nHowever, for Kubernetes Cluster administrators — it can soon become a nightmare if they give free reign to their application developers to use Host Path volumes — as can be seen from a recent conversation in the Kubernetes Slack.\n\n![slack conversation](/images/blog/slack-conversation.png) \n(***Issues - Slack conversation***)\n\nTypically Kubernetes Administrators lockdown (or disable) the `hostPath` feature using Pod Security Policies, especially in enterprises where security policies trump everything else.\n\nInterestingly, most of these challenges with the Host Path volumes can be overcome, by using a Dynamic Local PV Provisioner (managed by StorageClasses and PersistentVolumeClaims) that can be configured to create a Local PV with either a subpath or the entire disk.\n\nThe Kubernetes Cluster Administrators can reassert control by specifying via StorageClasses where and how the Host Path can be created and who can actually use them in the Cluster (by setting RBAC) policies.\n\nBesides overcoming the security issues with hostPath and gaining more control on how they are provisioned, Local PVs based provisioning of host paths has the following additional benefits:\n\n- Ability to pin the pods using the hostPath Local PV, by specifying the node selector. If native hostpath is used, there is no guarantee that the pod will get rescheduled to the same node in case of node reboots or pod eviction scenarios.\n- Ability to provide hooks that can either retain the PV and its data or perform clean up of the directory once the associated PVC is deleted.\n- Ability to provide exclusive ( ReadWriteOnce) or shared (ReadWriteMany) to the data on the host path.\n\nUsing Dynamic Provisioner for Local PVs also makes it easy to provide additional features of forthcoming Kubernetes Versions, such as:\n\n- Metrics Support\n- Enabling Capacity and PVC Resource Quotas\n- Full benefits of Volume Topology\n\n**Ready to give OpenEBS Local PVs a spin.**\n\n*OpenEBS Local PVs (alpha) is available in OpenEBS 0.9 release. Use the following commands to try it out.*\n\n*Step 1:* You need to have Kubernetes 1.12 or higher. Install the OpenEBS Local PV Provisioner.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.9.0.yaml\n\nNote: The above will setup the complete OpenEBS Control Plane along with the OpenEBS Dynamic PV Provisioner. Ensure that the ` openebs-localpv-provisioner` pod is running. `kubectl get pods -n openebs`\n\nThis version supports creating hostpath based Local PVs. A default StorageClass ` openebs-hostpath` is also loaded, that will create the host paths under `/var/openebs/local/`. If you would like to change the path, modify the StorageClass accordingly and re-apply.\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-hostpath\n annotations:\n openebs.io/cas-type: local\n cas.openebs.io/config: |\n - name: StorageType\n value: \"hostpath\"\n - name: BasePath\n value: \"/var/openebs/local/\"\n provisioner: openebs.io/local\n volumeBindingMode: WaitForFirstConsumer\n reclaimPolicy: Delete\n\n*Step 2:* Launch a MongoDB StatefulSet that makes use of Local Persistent Volumes.\n\n kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.9.x/k8s/demo/mongodb/demo-mongo-localpvhostpath.yaml\n\nCheckout that Local PVs are dynamically created — `kubectl get pv`.\n\nAlso go ahead and check that deleting the Pod will put it right back on the node where the PV was created.\n\n**Show me Code**\n\nIt is being cooked [here](https://github.com/openebs/maya/tree/master/cmd/provisioner-localpv). Any help with code review or suggestions welcome.\n\n**Credits!**\n\nThe implementation has been inspired by the feedback from the OpenEBS community who have requested for this feature to be available and used for Jiva Data engine and by all the prior work done by the Kubernetes Community.\n\n- [https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/tree/master/examples/hostpath-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/tree/master/examples/hostpath-provisioner)\n- [https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner](https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner)\n- [https://github.com/rancher/local-path-provisioner](https://github.com/rancher/local-path-provisioner)\n- [https://docs.openshift.com/container-platform/3.7/install_config/configuring_local.html](https://docs.openshift.com/container-platform/3.7/install_config/configuring_local.html)\n\n**What Next.**\n\nOpenEBS v1.0 continues to make significant progress in enhancing capabilities to the existing Storage Engines like ability to expand cStor Pools by adding additional disks, ability to replace a failed disk without affecting the applications, ability to resize the cStor volumes and many more. OpenEBS Local PVs will provide an additional option for the DevOps teams to cost effectively run their Stateful workloads.\n\nOpenEBS Dynamic Provisioner for Local Persistent Volumes works in tandem with the capabilities that are provided already by the OpenEBS Node Device Manager (NDM).\n\n![openebs local pv](/images/blog/openebs-local-pv.png)\n(***OpenEBS Local PVs***)\n\nThe Disk/Device Management Layer is provided by the OpenEBS NDM and projects built using Restic, that help with:\n\n- Discovering the devices attached to the Nodes\n- Performing Backup / Restore of Data from subpaths on the Nodes\n- Monitoring the devices used to provide the Host Paths, and in case the devices are backed by Disks — monitoring the SMART metrics to provide predictive error alerts.\n\nI believe this also opens up a lot of use cases to utilize the Capacity from the existing storage servers that are at the Users disposal by adding them to the Kubernetes Nodes and not having to worry about upgrading those SANs to support or wait for CSI specs to be implemented.\n\nWould love to hear your feedback on this feature. Do you use hostPaths or Local PVs that are natively supported in Kubernetes? What are your biggest challenges using those features and how can OpenEBS help? Please leave a comment in this blog or hit me up on the [Slack](https://slack.openebs.io/?__hstc=216392137.09f8198be7856f2c54869404c1891309.1580118253605.1580118253605.1580118253605.1&__hssc=216392137.1.1580118253605&__hsfp=3765904294).\n\n[Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":false,"slug":"dynamic-provisioning-of-kubernetes-local-pvs-using-openebs"},{"id":52,"title":"Block volume support for OpenEBS volume","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"27-03-2019","tags":["Block","Iscsi","Kubernetes","OpenEBS","Volume"],"excerpt":"A block volume is a volume that appears as a block device inside the container and allows low-level access to the storage without intermediate layers, as with file-system volumes.","content":"\nBy extending the API for *PersistentVolumes* to specifically request a raw block device, k8s provides an explicit method for volume consumption. Previously, any request for storage was always fulfilled using a formatted *filesystem*, even when the underlying storage was block storage. Kubernetes v1.13 moves raw block volume support to beta and allows persistent volumes to be exposed inside containers as a block device instead of as a mounted file system.\n\nIn addition, the ability to use a raw block device without a *filesystem* will give Kubernetes better support for high-performance applications that are capable of consuming and manipulating block storage for their needs. Block volumes are critical to applications such as databases (MongoDB, Cassandra) that require consistent I/O performance and low latency.\n\n## Block Volume Provisioning\n\nIn order to create a block volume, provisioners need to support *volumeMode*, then create a persistent volume with the desired volumeMode. If the admin selects an external provisioner that is capable of provisioning both filesystem and block volumes, he/she will have to carefully prepare the Kubernetes environment for their users as it is necessary for both Kubernetes itself and the external provisioner to support block volume functionality. Support for [block volume in Kubernetes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support) was introduced in v1.9 and promoted to beta in Kubernetes 1.13\n\nWith the release of OpenEBS 0.7.0, users can create a block volume via the OpenEBS external-provisioner. With this provisioner, during volume creation, the user requests the PersistentVolumeClaim, which sets *volumeMode=”Block”* in the *PersistentVolumeClaimSpec*, binds it with *PersistentVolume* objects, and Block devices are eventually attached to the pods by including them in the volumes array of the *podSpec.*\n\nRegardless of the *volumeMode*, provisioner can set *FSType* into the plugin’s volumeSource, but the value will be ignored at the volume plugin side if volumeMode = Block. Leaving *volumeMode* blank is essentially the same as specifying *volumeMode = “Filesystem,”* which results in the traditional behavior.\n\nWhen using a raw block volume in your Pods, you must specify a *VolumeDevice *attributein the Container section of the *PodSpec* rather than a *VolumeMount*. *VolumeDevices* utilize *devicePaths* instead of *mountPaths*. Inside the container, applications will see a device at that path instead of a mounted file system.\n\n## How to Use Block Volume\n\nThere are a number of use cases where using a raw block device can be useful. For example, A user can use a raw block device for database applications such as MySQL to read data from and write the results to a disk that has a formatted filesystem to be displayed via the ***nginx*** web server.\n\nThe following sections detail some sample volume specifications and steps to dynamically provision a raw block volume and attach it to an application pod.\n\n## Creating a new raw block PVC\n\nHere, the user creates one raw block volume and another formatted filesystem based volume that dynamically creates PersistentVolumes(PV).\n\n1. ***Raw Block volume:***\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-block-pvc\n spec:\n volumeMode: Block\n storageClassName: openebs-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\n2. ***Filesystem based volume:***\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol-pvc\n spec:\n storageClassName: openebs-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\n## Using a raw block PVC in POD\n\nHere, the user creates an application pod whose containers consume both block and filesystem volumes. We must choose devicePath for the block device inside the Mysql container rather than the mountPath for the file system.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: my-db\n spec:\n volumes:\n - name: my-db-data\n persistentVolumeClaim:\n claimName: demo-block-pvc\n - name: my-nginx-data\n persistentVolumeClaim:\n claimName: demo-vol-pvc\n containers\n - name: mysql\n image: mysql\n volumeDevices: \n - name: my-db-data\n devicePath: /var/lib/mysql/data\n - name: nginx\n image: nginx\n ports:\n - containerPort: 80\n volumeMounts:\n - mountPath: /usr/share/nginx/html\n name: my-nginx-data \n readOnly: false\n\n## Conclusion\n\nA *block volume* is a volume that appears as a block device inside the container and allows low-level access to the storage without intermediate layers, as with file-system volumes. There are several advantages *of raw disk partitions*, including:\n\n- Block devices that are actually SCSI disks support the sending of SCSI commands to the device using Linux ioctls.\n- Faster I/O without an overhead UNIX file system, more synchronous I/O without UNIX file system buffering, etc.\n\nThanks to [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page).\n","notHasFeatureImage":false,"slug":"block-volume-support-for-openebs-volume"},{"id":53,"title":"Litmus - Journey of a storage e2e and chaos framework for Kubernetes","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"26-03-2019","tags":["Chaos Engineering","Continuous Integration","Kubernetes","Litmus","OpenEBS"],"excerpt":"The objective of the Litmus project is to help evaluate stateful workloads on Kubernetes via storage & infrastructure focused end-to-end (e2e) tests that use chaos engineering tools & principles","content":"\n**This article is part of a #ProjectUpdates series on Litmus. In this blog post, we will share the latest developments in the project, which was announced & open-sourced during the [KubeCon Europe 2018](/blog/litmus-release-a-chaos-monkey-on-your-kubernetes-stateful-workloads)**\n\nThe objective of the Litmus project is to help evaluate stateful workloads on Kubernetes via storage & infrastructure focused end-to-end (e2e) tests that use chaos engineering tools & principles, amongst other aids, with the test cases executed as kubernetes jobs. Over the last 10 months, there have been several improvements to the framework based on community feedback alongside an increase in the test library. The focus areas & improvements through this period, ongoing efforts & upcoming features have been summarised in the subsequent sections.\n\n## Increased application & use-case coverage\n\nWhile the initial focus of Litmus centered around creating a K8s native approach to e2e (via test containers deployed as jobs) & facilitating a simpler interface for users to define & generate tests via English-like sentences (using godog), subsequent feedback from the extremely vibrant & involved OpenEBS community highlighted the need for users to be able to pick test cases, i.e., **deployable test jobs** from an existing test library to enable quick turnaround time for stateful workload evaluation. As a result, the existing ansible-based utils were extended.\n\nLitmus now includes support for multiple popular stateful applications (Percona MySQL, MongoDB, Apache Cassandra, MongoDB, Minio, Crunchy PostgreSQL, NuoDB, Prometheus etc..,) whilst covering common app lifecycle scenarios such as deployment, load, reschedule, scale-up, upgrade & de-provision. Also included are test jobs to verify the resiliency of these workloads against intermittent and permanent failures to persistent storage & cluster infrastructure, with these failures induced via chaos tools.\n\n## Refactoring Litmus into a more “microservice-oriented” model\n\nOne of the principles in the microservices world involves compartmentalization of an application service into several pieces to enable greater flexibility & control in administration. When the Litmus project was started, the entire workflow of the test: right from application deployment, through test business logic to clean up was embedded into a single job. While this continues to be a viable model in some circumstances, managing multiple user inputs/permutations at each stage of the workflow meant handling this in a single job would make the test a complex *“monolith”* & affect debuggability.\n\n![Continuous Integration (CI) pipelines](https://blog.mayadata.io/hs-fs/hubfs/Blog%20Images-60.png?width=834&name=Blog%20Images-60.png)Continuous Integration (CI) pipelines\n\nAnother consideration was Litmus tests being used to drive Continuous Integration (CI) pipelines that aim to simulate user environments, where applications are typically soaked over a period of time undergoing various *day-1/day-2-n* ops while being resilient to component failures. This resulted in the creation of different categories of Litmus jobs (henceforth interchangeably referred to as Litmusbooks), which can be combined in a specific order to achieve an “e2e test” or “Litmus experiment”.\n\n**App Deployers/De-provisioners:** Litmusbooks that take various parameters such as namespace, type, labels, liveness (IO) probes etc., to deploy or clean up the aforementioned applications.\n\n**Load generators:** Generates both synthetic (fio, sysbench, dd) & app-specific (TPC-C, YCSB) IO load on the applications using custom containers simulating client behavior.\n\n**External liveness monitors:** Monitors accessibility to an application server. While the standard K8s liveness/IO probes perform pod restarts to restore app server health, these external monitors test the seamless continuity of “*end-applications*” during the course of an app-lifecycle operation or chaos.\n\n**App/Storage lifecycle “operators”:** Performs standard functional operations such as app/storage scale, backup/restore, reschedule, app/storage upgrades etc.., (should not be mistaken for actual “kubernetes operators”)\n\n**Chaos inducers:** Litmusbooks that induce failures on specified targets on the cluster, including application replicas, storage & cluster infrastructure (network, nodes, disks, memory, CPU)\n\n![Litmus Framework](/images/blog/litmus-framework.png)\n\n(***Litmus Framework***)\n\nTypically, the lifecycle functionality & chaos litmusbooks include steps for post-test health checks on the application and persistent storage components, including **data integrity** validation. All the above requirements necessitated the creation of multiple “*test facilitator containers*”(maintained [here](https://github.com/openebs/test-tools)) that implement python-based application clients and integrate other popular opensource chaos tools such as **pumba**, **chaostoolkit** & **chaoskube**.\n\nFor more details on chaos engineering using Litmus, read this [article](/blog/storage-chaos-engineering-with-litmus-an-overview?__hstc=216392137.a909c2d2cc44264c6d1aa717b549f14e.1580153178674.1580153178674.1580153178674.1&__hssc=216392137.1.1580153178674&__hsfp=3765904294).\n\n## Litmus e2e test result visualization\n\nOne of the important enablers for the adoption of any given test (suite) into a CI framework is the result mechanism to determine success/failure. During the said period, Litmus moved over from extracting results from test container logs to storing this information in a Kubernetes static custom resource (*“litmusresult”, which is updated by the test code at the beginning & end of test execution via a JSON merge patch operation*), with the resource name dynamically generated based on user tags. This resource also holds other test metadata such as *appType*, *testType*, *testStatus* etc., apart from the actual *verdict* (result). CI jobs that implement/run the litmusbooks can be configured to derive job status from these CRs.\n\n![cr-structure](/pubic/images/blog/cr-structure.png)A jinja template describing the result CR structure\n\n## Other notable enhancements\n\nThe project also saw some other organic changes which have resulted in its faster adoption into CI systems & increased efficiency of the tests. They have been briefly described below:\n\n**Infra Setup Playbooks**: Litmus now provides multiple ansible playbooks to aid quick and easy creation of Kubernetes clusters on different popular cloud platforms (GCP, AWS, GKE, AKS, EKS, Packet Cloud, On-premise vSphere VMs, OpenShift-on-AWS, OpenShift-on-vSphere) with minimal config inputs & also litmusbooks to setup the OpenEBS control plane on a K8s cluster.\n\n**Simplifying Litmus Setup**: The litmus pre-requisites are taken care of by a simple interactive bash script that sets up the RBAC, CRDs & config file updates.\n\n**K8s-Specific Enhancements**: These include small changes to make the litmusbooks more robust, such as:-\n\n- Usage of generateName API in the litmus jobs to ensure repeated execution doesn’t cause failures\n- Maximum Usage of labels & JSONPATH expressions in the kubectl commands used in the test code instead of bash utilities to guard against K8s cluster version & storage resource changes.\n- Increased use of pod ENV variables (user-specific & downward API) to control test inputs in a K8s-native way rather than pass them in a test-code specific form (ansible extra-vars)\n\n**Increased Modularization of Test Code**: Most of the test code is reused across the litmusbooks, due to which reusable tasks are managed as separate utils. These are further classified as *common*, *chaos* & *app/storage lifecycle* modules, with well-defined interfaces.\n\n**Litmus CI**: The Litmus (and test-tools) project now consists of its own Travis-based CI which builds respective runner images post-Ansible/Python lint/syntax check operations.\n\n**Optimized Test Images**: The litmus runner (go, ansible) as well the test facilitators images have been optimized with respect to the number of layers, size and binary versions used.\n\n## Case Study: OpenEBS CI Powered By Litmus\n\n![openebs-ci-powered-by-litmus](/images/blog/openebs-ci-powered-by-litmus.png)\n\n(***OpenEBS CI Powered By Litmus***)\n\nThe Continuous Integration framework for the OpenEBS project, i.e., the various components such as Maya, Node-Disk-Manager (control plane), Jiva, cStor (data plane) is built using **Gitlab**, with the e2e tests executed via **Litmusbooks** & supported by an Elasticsearch-Fluentd-Kibana (**EFK**) based logging framework. Post the build phase (where the above-mentioned components are built and images pushed to respective repositories) multiple gitlab e2e pipelines based on the Kubernetes cluster version are triggered, each of which includes the following litmus-powered stages to:\n\n- Create K8s cluster on the bare metal cloud platform Packet\n- Deploy the OpenEBS control plane\n- Deploy stateful applications\n- Run the app & storage lifecycle ops\n- Induce application, OpenEBS persistent storage & cluster infrastructure Chaos\n- De-provision the applications & OpenEBS control plane\n- Delete the K8s cluster resources\n\nThe **gitlab runner** scripts follow a defined pattern that involves the litmusbook preconditioning (user inputs), followed by litmusbook execution & monitoring of result custom resources, from which the success of gitlab job (and hence the stage) is derived.\n\n[openebs.ci](https://openebs.ci/) *provides a dashboard view of the build status for various commits to the openebs control plane and data plane projects, with detailed visibility into individual stage/job status & respective logs on Kibana. It also provides a real-time view of long-running production workloads using OpenEBS PVs which are maintained over multiple OpenEBS releases, with an option to induce random chaos on application/storage components & view the behavior. Litmus chaos modules are called internally to achieve this.*\n\nMore details on the composition & functioning of this CI framework is available in this [blog](/blog/a-primer-on-openebs-continuous-integration?__hstc=216392137.a909c2d2cc44264c6d1aa717b549f14e.1580153178674.1580153178674.1580153178674.1&__hssc=216392137.1.1580153178674&__hsfp=3765904294) post.\n\n## Ongoing Efforts & Upcoming Features In Litmus\n\nAt this point, the Litmus contributions are mainly focused towards increasing the application & storage life cycle tests, more specifically, **Day-2 operations** of standard stateful applications, while continuing to increase the **stability** of the current tests.\n\nSome of the near-term improvements & enhancements include:\n\n- Publish documentation for the Litmus project\n- Improvement of the Litmus Result CR to hold real-time status with increased metadata on failures, including the ability to capture application state.\n- Development of additional infrastructure & application-specific chaos utils\n\n## Conclusion\n\nThe improvements made to the Litmus project have been possible only due to the amazing support from the community both in terms of feedback, as well as contributions. So, a huge thank you & looking forward to more of the same!!\n","notHasFeatureImage":false,"slug":"litmus---journey-of-a-storage-e2e-and-chaos-framework-for-kubernetes"},{"id":54,"title":"(Storage) Chaos Engineering with Litmus - An Overview","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"20-03-2019","tags":["Chaos Engineering","Docker","Kubernetes","Litmus","OpenEBS"],"excerpt":"In this extremely insightful article, Evan Powell outlines various possible failures that could occur in a Kubernetes cluster running stateful workloads & the possible means to mitigate them..","content":"\n**This article is part of a #Concepts series on Kubernetes and Litmus. Here, we present an overview of why & how the Litmus project employs chaos techniques to test the resiliency of OpenEBS storage solutions.**\n\nIn this extremely insightful [article](/blog/ha-vs-dr-and-ha-for-your-db), [Evan Powell](https://medium.com/u/36584a5b84a) outlines various possible failures that could occur in a Kubernetes cluster running stateful workloads & the possible means to mitigate them, while advocating Chaos Engineering as a means to gauge system behavior, blast radius & recovery times. While there are some amazing chaos tools in the Kubernetes ecosystem, both opensource ([chaoskube](https://github.com/linki/chaoskube), [pumba](https://github.com/alexei-led/pumba), [chaostoolkit](https://github.com/chaostoolkit/chaostoolkit)etc.,) & lately, free solutions ([gremlin-free](https://www.gremlin.com/blog/introducing-gremlin-free/)) which can help induce specific types of failures on containers, pods, virtual machines and cloud instances, a need was identified to tie the act of chaos itself with **automated orchestration** (steady state condition checks) & **analysis** (data availability & integrity, application & storage deployments’ health etc.,).\n\nLitmus was [developed ](https://medium.com/@karthik.s_5236/litmus-journey-of-a-storage-e2e-and-chaos-framework-for-kubernetes-dc09a3904a24)in order to (amongst other needs) facilitate a Kubernetes-native & persistent storage, i.e., stateful workload-focused approach to chaos, with ability to self-determine the success of a controlled chaos experiment based on pre-defined exit criteria. In other words, a framework to construct & host a ready set of “deployable tests” that use [chaos engineering principles](https://principlesofchaos.org/) as the underlying business-logic. In order to achieve this, Litmus internally makes use of the aforementioned opensource chaos tools, combined with the power of kubectl, while also employing certain home-grown chaos techniques where necessary.\n\nThough Litmus has been designed to be easily extendable to work with different persistent storage solutions, the [OpenEBS ](https://openebs.ci/)community is the primary consumer of these tests. As a result, the majority of the utils/modules are chaos functions directed against the OpenEBS storage (persistent volume) resources (target pod, replica pod, storage pools etc.,) and their corresponding state checks. There are, however, app-specific and general infrastructure chaos tests too.\n\n**T**he following sections will briefly discuss the approaches and patterns used in the design of the different litmus chaos tests. The objective here is to introduce the larger community to methods being followed in this project and receive valuable feedback against them! The step-by-step illustration on running these tests, with details on impact/behavior with respect to different OpenEBS storage engines, will be covered in separate blog posts in the #HowDoI series.\n\n**The Litmus Job Template**\n\n***Note**: _To set the context (though it may already be known), a majority of the litmus tests are essentially ansible playbooks being executed from within a runner pod, which in turn is managed by the Kubernetes job controller._*\n\nOne of the requirements of the “deployable” litmus test or chaos experiment is that the *control variables* (typically, chaos params — considering that different applications, storage engines, PV component versions, and deployment models are evaluated against a particular chaos sequence) and *independent variables* (PV, application and K8s deployment params) should be defined in the test artifact, which, in this case, is a Kubernetes job specification. This is achieved by specifying them as pod/container environment (ENV) variables. In most cases, the final object (cluster resource) of chaos is derived from the application or persistent volume claim info passed in the job.\n\n![sample litmus chaos test](/images/blog/sample-litmus-chaos-test.png)\n\n(***Sample Litmus Chaos Test Variables/Tunables***)\n## Test Execution Workflow\n\nThe standard chaos test execution workflow is described in the schematic diagram shown below.\n\n![litmus chaos test execution](/images/blog/litmus-chaos-test-execution.png)\n\n(***Litmus Chaos Test Execution Workflow***)\n## Container Crash\n\nLitmus makes use of a [Pumba daemonset](https://github.com/openebs/litmus/blob/master/chaoslib/pumba/pumba_kube.yaml) to set up the chaos infrastructure, as this is necessary for the target and pumba containers to co-exist on a node so the latter can execute the Kill operation on the target. These pods are configured to run in --dry mode and have the host’s /var/run/docker.sock mounted as a volume, thereby allowing the test to launch custom chaos commands using the *pumba-cli* through kubectl exec operations.\n\nPumba “kills” a container by terminating the main process on the target container, and multiple termination signals (SIGKILL, SIGTERM, SIGSTOP)are supported. The object to be killed is either the app or PV target/replica/pool containers or their sidecars.\n\nA sample chaos util (essentially, a task file that is imported/called in the main chaos test playbook) for performing a container kill can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/pumba/pod_failure_by_sigkill.yaml).\n\n## Forced Pod Reschedule\n\nPod deletion, which is managed by higher level kubernetes controllers such as deployments and statefulsets, causes the pod to be rescheduled on a suitable node (same or different) in the cluster. Continuity of end application I/O and the time taken for reschedule/bring-up are the crucial post checks performed in these tests.\n\nLitmus uses a [chaoskube deployment](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/chaoskube.yaml) (setup with RBAC) configured in a default --dry-run mode, allowing the test to run a custom pod failure chaos command using the *chaoskube-cli* through kubectl exec operations. This is done by providing appropriate label and namespace filters to point to the desired target pods.\n\n*Most of the chaos tests confirm the success of chaos induction (which itself “can” fail) by observing an increment in the *resourceVersion* of the target deployments (component under test) as part of the post-chaos checks.*\n\nA sample chaos util for performing a forced pod reschedule using chaoskube can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/pod_failure_by_chaoskube.yaml).\n\n## Network Delay and Packet loss to Specific Storage Resources\n\nLitmus uses the Pumba daemonset (which internally makes use of [tc ](https://linux.die.net/man/8/tc)and [netem](http://man7.org/linux/man-pages/man8/tc-netem.8.html) tools on the target containers) and tc directly in some cases to generate *egress delays* and *percentage packet loss* on the specified targets for the desired duration. Pumba also has the capability to induce network delays using random distribution models (uniform, normal, Pareto, Paretonormal) and loss models (Bernoulli, Markov, Gilbert-Elliott).\n\nThese capabilities are used to simulate flaky iSCSI storage targets, lousy data-replication channels, and test data reconstruction post temporary component loss (such as storage replicas and pools).\n\nSample chaos utils for inducing network delays and packet loss can be viewed [here ](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/cstor_target_network_delay.yaml)and [here,](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml)[https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml](https://github.com/openebs/litmus/blob/master/chaoslib/openebs/inject_packet_loss_tc.yml)respectively.\n\n## Forced Reschedule via Taint-Based Evictions\n\nUnder certain specific node conditions/states, Kubernetes automatically taints the node with what is commonly referred to as “*eviction taints*”. These have a “*NoExecute*” taint effect. This means the pods scheduled and running on these nodes are immediately evicted and rescheduled onto other suitable nodes on the cluster. Amongst these taints, Kubernetes automatically adds a default toleration period *(tolerationSeconds)* of 300s for **node.kubernetes.io/unreachable** & **node.kubernetes.io/not-ready** to account for transient conditions. For other taints, such as **node.kubernetes.io/memory-pressure**, **node.kubernetes.io/disk-pressure,** etc.., the pods are evicted immediately (unless specifically tolerated in the deployment spec. Visit [kubernetes docs](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) for more details).\n\nThis scenario holds relevance in the case of stateful workloads, where an *en-masse* movement of the application and storage resources (which may include control plane components) can have potential data availability considerations. In other words, it is an effective HA test, not dissimilar to a graceful node loss.\n\nLitmus uses **kubectl** to simulate such node conditions by manually applying the taint on desired nodes (typically, the one which hosts the application and storage target pods) while monitoring end-application I/O. A sample chaos util for eviction by taints can be viewed [here](https://github.com/openebs/litmus/blob/master/chaoslib/kubectl/pod_evict_by_taint.yaml). Litmus simulates graceful node loss via another mechanism: **Cordon and Drain** (this also triggers reschedule of all the node’s pods), which is typically used in cluster maintenance workflows. This util is available [here](https://github.com/openebs/litmus/blob/master/chaoslib/kubectl/cordon_drain_node.yaml)**.**\n\n## Simulated Daemon Service Crashes\n\nLoss of node services (*kubelet*, *docker*) is one of the most common reasons for failed/non-starting pods and inaccessible application/storage services on the cluster. Litmus uses a [privileged daemonset](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/svc_chaos_kube.yml) that mounts the host’s root filesystem paths in order to execute systemctl commands on the desired node (again, typically the node running application and storage target pods).\n\nIn case of managed (GKE) or cloud-hosted (KOPS based AWS, for ex.) kubernetes clusters, these crucial daemon services often have corresponding health check/monitoring services that detect anomalies and attempt a restart to restore node health. The litmus chaos utils disable these monitoring services temporarily for the duration of the chaos to ensure consistent chaos rather than flaky state changes (this, it can be argued, is a case for a separate test!). After the chaos duration has elapsed, all affected services are restored, with post checks confirming cluster health.\n\nThese scenarios cause the ungraceful/forced reschedule workflows in Kubernetes to kick-in. They have significant implications on stateful workloads, with disk/volume unmount-detach and reattach-remount times in focus. As with the eviction by taint and drain scenarios, Litmus monitors end-application I/O and data integrity as part of the chaos experiment.\n\nSample utils for service chaos on GKE and AWS clusters can be viewed [here ](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/svc_chaos.yml)and [here](https://github.com/openebs/litmus/blob/master/chaoslib/service_chaos/aws-service.yml)**,** respectively.\n\n## Resource Exhaustion on Kubernetes Nodes\n\nWhile we discussed simulating node conditions via eviction taints, it is also necessary to actually induce resource exhaustion such as *CPU burn*, *Memory Hog*, and *Filled Disks* to gauge real-time behavior. Litmus uses simple docker containers that run (a) a python program to keep appending large data to a growing string to [consume system memory](https://github.com/openebs/test-tools/tree/master/memleak), and (b) a forkbomb used to cause process multiplication resulting in [CPU burn](https://github.com/openebs/test-tools/tree/master/forkbomb).\n\n![system memory consumption](/images/blog/system-memory-consumption.png)(a) \n(***system memory consumption***)\n\n![cpu burn](/images/blog/cpu-burn.png)(b) \n(***CPU burn via forkbomb***)\n\nA [node-action daemonset](https://github.com/openebs/litmus/blob/master/apps/percona/chaos/node_freeze/node_freeze.yml) that mounts the host’s **/var/run** is used to execute docker commands that run the above containers with Litmus monitoring node status and end-application I/O. The sample chaos util for resource exhaustion can be found [here](https://github.com/openebs/litmus/blob/master/chaoslib/chaoskube/node_freeze_chaos.yml)**.**[Disk/Capacity fill utils](https://github.com/openebs/litmus/tree/master/apps/fio/tests/pool_capacity), available in Litmus currently, are implemented within the storage volume and storage pool layers using **fio** tool.\n\n## Kubernetes Node (instance) Failures\n\nLitmus supports instance termination on AWS (KOPS based Kubernetes clusters) and Packet (Bare-metal cluster using kubeadm). Work is in progress to extend the similar capabilities to other platforms as well. The different impact is noticed when node failures are performed across platforms. For example, in the case of AWS, another instance with the same root disk is automatically brought up as the nodes are by default a part of the KOPS auto-scaling instance group. With Packet, however, the node remains shut down and can be subsequently powered-on. These behaviors expose different workflows of the underlying storage solution.\n\nFor AWS, Litmus uses [chaostoolkit templates ](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/random_node_terminate_aws.j2)that are executed in a [Kubernetes job](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/node-stop.yml)**,** which in turns runs a [custom container](https://github.com/openebs/test-tools/blob/master/chaostoolkit-aws/Dockerfile) using chaostoolkit binary and is equipped with the desired libraries. The AWS access keys and instance details are passed as test inputs from the Litmus test job. The util is capable of random instance selection in a given *Availability Zone*, as well as termination of a specific instance.\n\nFor Packet, Litmus uses the [Ansible Packet module](https://docs.ansible.com/ansible/latest/modules/packet_device_module.html#packet-device-module) **to power-off** and power-on the node instance, with the packet *API keys*, *project id*, *node name* and *chaos duration* passed as test inputs.\n\nThe AWS and Packet node failure chaos utils can be found [here ](https://github.com/openebs/litmus/blob/master/chaoslib/chaostoolkit/aws/chaostoolkit_util.yml)and [here](https://github.com/openebs/litmus/blob/master/chaoslib/packet/node_chaos.yml).\n\n## Disk Failures\n\nLitmus currently supports disk failure chaos utils for Kubernetes clusters built on an AWS cloud. Work is in progress to extend this to other platforms. The AWS CLI and [Boto Python SDK](https://boto3.readthedocs.io/) are used to force detach and reattach the EBS volumes used as the disk sources for the storage pools. As with other chaos tests, Litmus monitors the end-application I/O and data integrity over the course of the experiment.\n\nA Sample EBS disk failure util can be found [here](https://github.com/openebs/litmus/blob/master/chaoslib/aws_chaos/disk_failure.yml)**.**\n\n## Conclusion\n\nAt this point in Litmus’ journey, improving the automated analysis capabilities, increasing the chaos function coverage across components, creating greater support for common stateful applications, and enhancing the stability of existing chaos tests are the top priorities. However, as happy members of the chaos engineering community, we are always on the lookout for updates and new entrants in the ecosystem and any other ways we can contribute! We hope this article has been helpful in gaining a better understanding of how Litmus does chaos. As mentioned earlier, we always look forward to your feedback & comments!!\n","notHasFeatureImage":false,"slug":"(storage)-chaos-engineering-with-litmus---an-overview"},{"id":55,"title":"PostgreSQL anywhere — via Kubernetes with some help from OpenEBS and credativ engineering","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"18-03-2019","tags":["DevOps","Kubernetes","Postgres","Postgresql","Solutions"],"excerpt":"In this blog, we’d like to briefly cover how using cloud-native or “container attached” storage can help in the deployment and ongoing operations of PostgreSQL on Kubernetes.","content":"\nby [Murat Karslioglu](https://twitter.com/muratkarslioglu), [OpenEBS](https://openebs.io/) and [Adrian Vondendriesch](https://twitter.com/__discostu__), [credativ](https://www.credativ.com)\n\n### Introduction\n\nIf you are already running Kubernetes on some form of cloud whether on-premises or as a service, you understand the ease-of-use, scalability and monitoring benefits of Kubernetes — and you may well be looking at how to apply those benefits to the operation of your databases.\n\nPostgreSQL remains a preferred relational database, and although setting up a highly available Postgres cluster from scratch might be challenging at first, we are seeing patterns emerging that allow PostgreSQL to run as a first class citizen within Kubernetes, improving availability, reducing management time and overhead, and limiting cloud or data center lock-in.\n\nThere are many ways to run high availability with PostgreSQL; for a list, see the [PostgreSQL Documentation](https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling). Some common cloud-native Postgres cluster deployment projects include [Crunchy Data](https://www.crunchydata.com/)’s, [Sorint.lab](https://www.sorint.it/)’s [Stolon](https://github.com/sorintlab/stolon) and [Zalando](https://jobs.zalando.com/tech/)’s [Patroni](https://github.com/zalando/patroni)/[Spilo](https://github.com/zalando/spilo). Thus far we are seeing Zalando’s operator as a preferred solution in part because it seems to be simpler to understand and we’ve seen it operate well.\n\nSome quick background on your authors:\n\n- OpenEBS is a broadly deployed OpenSource storage and storage management project sponsored by MayaData.\n- [credativ](https://www.credativ.com) is a leading open source support and engineering company with particular depth in PostgreSQL.\n\n> **In this blog, we’d like to briefly cover how using cloud-native or “container attached” storage can help in the deployment and ongoing operations of PostgreSQL on Kubernetes. This is the first of a series of blogs we are considering — this one focuses more on *why* users are adopting this pattern and future ones will dive more into the specifics of *how* they are doing so.**\n\n> **At the end you can see how to use a Storage Class and a preferred operator to deploy PostgreSQL with OpenEBS underlying**\n\nIf you are curious about what container attached storage of CAS is , you can read more from the Cloud Native Computing Foundation (CNCF) here:\n\n[https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/)\n\nConceptually, you can think of CAS as the decomposition of previously monolithic storage software into containerized microservices that run on Kubernetes. This provides the same advantages of running Kubernetes to the storage and data management layer as well. Like Kubernetes, OpenEBS runs anywhere, so the same advantages below apply whether it is used on-premises or on any of the hosted Kubernetes services.\n\n**PostgreSQL plus OpenEBS**\n\n![Postgres Operator using OpenEBS](/images/blog/postgres-operator-using-openebs.png)\n(***Postgres-Operator using OpenEBS as the Storage Provider***)\n\nWe have seen joint users adopting OpenEBS as a substrate to PostgreSQL for a variety of reasons. Here are a few of the reasons that stand out:\n\n**Consistency in underlying disk or cloud volume management.**\n\nOne of the most annoying things about setting up a system to run PostgreSQL, even if it is on Kubernetes, is configuring the underlying disks and storage systems as needed. With a solution such as OpenEBS, you specify how you want the underlying systems configured via storage classes. OpenEBS, with the help of Kubernetes, ensures that the system delivers the required storage capacity and is configured as you need. An example of such a storage class is shared below. This automation can remove a source of human error and annoyance.\n\n**Thin provisioning and on-demand expansion.**\n\nNow that you have turned over to OpenEBS and the provisioning and management of the underlying storage hardware and services, you simply tell it the amount of storage you need for your PostgreSQL and everything will work, right? In reality, knowing how much data your PostgreSQL instance or instances will consume is difficult and arguably impossible as it is beyond your control.\n\nOpenEBS can also help here as it supports both thin provisioning and on-the-fly pool expansion. The thin provisioning allows you to claim more space than you actually can provision, allowing your PostgreSQL to scale in space usage without interruption. This is done by allowing the addition of more storage to the running system without the need to stop the database.\n\nThin provisioning, however, is not a good idea if there is no on-the-fly expansion of the underlying capacity for obvious reasons: as the PostgreSQL expands, you want to ensure it can claim space as needed. Otherwise, at some point you will have to interrupt operations and again perform manual tasks. OpenEBS helps here as well — if configured to do so, it can expand its underlying pools, whether these are of physical disks, underlying storage systems, or storage services from a cloud. The capacity of the pool can be expanded on demand by simply adding more disks to the cStor pool.\n\nThe cStor architecture also supports the resizing of a provisioned volume on the fly, which will be fully automated as of OpenEBS 0.9. With these enhancements, volumes and underlying pools will be able to scale automatically on any cloud providing K8s support.\n\nIn addition to reducing the risk and hassle of manual operations, the combination of thin provisioning and on-demand scaling can reduce costs. This is the case because you don’t over-provision capacity to achieve performance, which reduces unnecessary cloud service spending and can increase average utilization of hardware usage as well.\n\n**Disaster recovery and migration.**\n\nWith a solution like OpenEBS, your storage classes can also include backup schedules. These can be easily managed via Kubectl, which is free to use. Again, these storage classes can be applied on a per-container basis, which offers quite a bit of granularity and control by each team running their PostgreSQL.\n\nAdditionally, we are working together to increase integration with PostgreSQL to this per-snapshot based workload, per container backup capability, called DMaaS by MayaData and OpenEBS. With this additional integration, an option will be added to the storage classes and to OpenEBS that flushes active transactions before taking the snapshot. The additional integration of storage snapshots in conjunction with Write Ahead Log (WAL) archiving will provide additional PITR functionality. DMaaS leverages the open-source Velero from Heptio and marries it to the COW based capabilities of the cStor OpenEBS engine to deliver highly-efficient backups and migrations.\n\nWith DMaaS, backups that are taken to one location can be recovered from another. This can be very useful for a variety of use cases, including the use of relatively ephemeral clusters as a part of a rolling upgrade of an environment. Additionally, the same capability can be used to move workloads from one Kubernetes environment to another, thereby reducing lock-in.\n\n**Snapshots and clones for development and troubleshooting.**\n\nDBAs have long been using snapshots and clones to assist in troubleshooting and allow teams to develop and test against a read-only copy of production data. For example, with OpenEBS you can easily use Kubernetes to invoke a snapshot, then promote that snapshot to a clone and spawn a container from that clone. You now have the ability to do anything you want with that container and the contained dataset, and can destroy it when you are finished.\n\nOne use case that clones can support is improved reporting. For example, let’s say you use computationally expensive analytical queries and build roll-up queries for monthly reports. With OpenEBS, it is simple to clone the underlying OLTP system, therefore allowing you to work on a static copy of your database and remove load from your production DBs. This ensures you have a verifiable source of information for those reports.\n\n**Running Postgres-Operator on OpenEBS**\n\n![PostgreSQL with OpenEBS persistent volumes](/images/blog/postgres-operator-on-openebs.png)(***PostgreSQL with OpenEBS persistent volumes***)\n\n**Software Requirements**\n\n- [Postgres-Operator](https://github.com/zalando/postgres-operator) (for cluster deployment)\n- [Docker](https://docs.docker.com/install/)\n- Kubernetes 1.9+\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- [OpenEBS](https://github.com/openebs/openebs)\n\n**Installing OpenEBS**\n\nIf OpenEBS is not installed in your K8s cluster, you can do it [here](/docs/user-guides/installation?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025).\n\n**Configure cStor Pool**\n\n1. If cStor Pool is not configured in your OpenEBS cluster, follow the steps presented [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025). As PostgreSQL is a StatefulSet application, it requires a single storage replication factor. If you prefer additional redundancy, you can always increase the replica count to 3.\nDuring cStor Pool creation, make sure that the maxPools parameter is set to >=3. If a cStor pool is already configured, move on to the next step. A sample YAML named openebs-config.yaml can be used for configuring cStor Pool and is provided in the Configuration details below.\n\n**openebs-config.yaml**\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE — Appropriate disks need to be fetched using `kubectl get disks`\n #\n # `Disk` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n # Replace the following with actual disk CRs from your cluster `kubectl get disks`\n # Uncomment the below lines after updating the actual disk names.\n disks:\n diskList:\n # Replace the following with actual disk CRs from your cluster from `kubectl get disks`\n # — disk-184d99015253054c48c4aa3f17d137b1\n # — disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # — disk-806d3e77dd2e38f188fdaf9c46020bdc\n # — disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # — disk-bad1863742ce905e67978d082a721d61\n # — disk-d172a48ad8b0fb536b9984609b7ee653\n — -\n\n**Create the Storage Class**\n\n1. You must configure a StorageClass to provision a cStor volume on a cStor pool. In this solution, we use a StorageClass to consume the cStor Pool. This is created using external disks attached on the Nodes. The storage pool is created using the steps provided in the [Configure StoragePool](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025) section. In this solution, PostgreSQL is a deployment. Because this requires replication at the storage level, the cStor volume replicaCount is 3. A sample YAML named openebs-sc-pg.yaml used to consume the cStor pool with a cStorVolume Replica count of 3 is provided in the configuration details below.\n\n**openebs-sc-pg.yaml**\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-postgres\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"3\" \n provisioner: openebs.io/provisioner-iscsi\n reclaimPolicy: Delete\n ---\n\n**Launch and Test Postgres Operator**\n\n1. Clone Zalando’s Postgres Operator.\n\n git clone https://github.com/zalando/postgres-operator.git\n cd postgres-operator\n\n**Use the OpenEBS Storage Class**\n\n1. Edit the manifest file and add openebs-postgres as the storage class.\n\n nano manifests/minimal-postgres-manifest.yaml\n\nAfter adding the storage class, it should look similar to the example below:\n\n apiVersion: \"acid.zalan.do/v1\"\n kind: postgresql\n metadata:\n name: acid-minimal-cluster\n namespace: default\n spec:\n teamId: \"ACID\"\n volume:\n size: 1Gi\n storageClass: openebs-postgres\n numberOfInstances: 2\n users:\n # database owner\n zalando:\n - superuser\n - createdb\n # role for application foo\n foo_user: []\n #databases: name->owner\n databases:\n foo: zalando\n postgresql:\n version: \"10\"\n parameters:\n shared_buffers: \"32MB\"\n max_connections: \"10\"\n log_statement: \"all\"\n\n**Start the Operator**\n\n1. Run the following command to start the operator:\n\n kubectl create -f manifests/configmap.yaml # configuration\n kubectl create -f manifests/operator-service-account-rbac.yaml # identity and permissions\n kubectl create -f manifests/postgres-operator.yaml # deployment\n\n**Create a Postgres Cluster on OpenEBS (Optional)**\n\nThe operator can run in a namespace other than default. This may be done to use the test namespace, for example. Run the following before deploying the operator’s manifests:\n\n kubectl create namespace test\n kubectl config set-context $(kubectl config current-context) — namespace=test\n\n1. Run the command below to deploy from the example manifest:\n\n kubectl create -f manifests/minimal-postgres-manifest.yaml\n\n2. It only takes a few seconds to get the persistent volume (PV) for the *pgdata-acid-minimal-cluster-0* up and running. Check the PVs created by the operator using the k*ubectl get pv* command:\n\n $ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-8852ceef-48fe-11e9–9897–06b524f7f6ea 1Gi RWO Delete Bound default/pgdata-acid-minimal-cluster-0 openebs-postgres 8m44s\n pvc-bfdf7ebe-48fe-11e9–9897–06b524f7f6ea 1Gi RWO Delete Bound default/pgdata-acid-minimal-cluster-1 openebs-postgres 7m14s\n\n**Connect to the Postgres Master and Test**\n\n1. If not installed previously, install psql client using this command:\n\n sudo apt-get install postgresql-client\n\n2. Run the command below and note the hostname and host port.\n\n kubectl get service — namespace default |grep acid-minimal-cluster\n\n3. Next, run the following commands to connect to your PostgreSQL DB and test. Replace the *[HostPort]* below with the port number from the output of the above command:\n\n export PGHOST=$(kubectl get svc -n default -l application=spilo,spilo-role=master -o jsonpath=\"{.items[0].spec.clusterIP}\")\n export PGPORT=[HostPort]\n export PGPASSWORD=$(kubectl get secret -n default postgres.acid-minimal-cluster.credentials -o ‘jsonpath={.data.password}’ | base64 -d)\n psql -U postgres -c ‘create table foo (id int)’\n\nCongratulations! You now have the Postgres-Operator and your first test database operating with the help of cloud-native OpenEBS storage.\n\n**Partnership and Future Direction**\n\nAs this blog indicates, the teams at MayaData/OpenEBS and credativ are increasingly working together to offer more benefits for organizations running PostgreSQL and other stateful workloads. In future blogs, we’ll provide more hands-on tips.\n\nWe are certainly looking for feedback and suggestions on the strategic direction of this collaboration. Please provide feedback below or find us on [Twitter](https://twitter.com/openebs) or the OpenEBS slack community: [http://slack.openebs.io](http://slack.openebs.io/?__hstc=216392137.adc0011a00126e4785bfdeb5ec4f8c03.1580115966430.1580115966430.1580115966430.1&__hssc=216392137.1.1580115966431&__hsfp=818904025)\n","notHasFeatureImage":false,"slug":"postgresql-anywhere — via-kubernetes-with-some-help-from-openebs-and-credativ-engineering"},{"id":56,"title":"How to deploy NuoDB on Kubernetes with OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"14-03-2019","tags":["DevOps","Kubernetes","Nuodb","Solutions","OpenEBS"],"excerpt":"NuoDB is a container-native adaptive-scale distributed SQL database designed with distributed application deployment challenges in mind.","content":"\n**NuoDB** is a container-native adaptive-scale distributed SQL database designed with distributed application deployment challenges in mind. The NuoDB database is ANSI SQL compliant and seen in a new category called “NewSQL.” If you are not familiar, what is different with NewSQL solutions is explained in a 2016 research paper [here](https://sigmodrecord.org/publications/sigmodRecord/1606/pdfs/07_forum_Pavlo.pdf).\nSome of the solutions in this category are [Amazon Aurora](https://en.wikipedia.org/wiki/Amazon_Aurora), [Google Spanner](https://en.wikipedia.org/wiki/Google_Spanner), TiDB, [CockroachDB](https://en.wikipedia.org/wiki/Cockroach_Labs), [Altibase](https://en.wikipedia.org/wiki/Altibase), [Apache Ignite](https://en.wikipedia.org/wiki/Apache_Ignite), [GridGain](https://en.wikipedia.org/wiki/GridGain_Systems), [Clustrix](https://en.wikipedia.org/wiki/Clustrix), [VoltDB](https://en.wikipedia.org/wiki/VoltDB), [MemSQL](https://en.wikipedia.org/wiki/MemSQL), [NuoDB](https://en.wikipedia.org/wiki/NuoDB), [HarperDB](https://en.wikipedia.org/w/index.php?title=HarperDB&action=edit&redlink=1), and [Trafodion](https://en.wikipedia.org/wiki/Trafodion)\n\nNuoDB has a distributed object architecture that works in the cloud, which means that in order to scale-out the database, a new node can be added and database runs faster. That’s why it represents the perfect example of a new-gen cloud-native architecture and a solution for the containerized applications running on Kubernetes.\n\nNuoDB provides all the properties of ACID-compliant transactions and standard relational SQL language support. It’s designed from the start as a distributed system that scales the way a cloud-native service is expected to scale, providing **continuous availability (CA)** and resiliency with no single points of failure. Different from traditional shared-disk or shared-nothing architectures, NuoDB presents a new kind of peer-to-peer, on-demand independence that yields CA, low-latency, and a deployment model that is easy to manage. The chart below is a simple comparison and decision chart from the [*NuoDB website*](https://www.nuodb.com/product/database-comparison).\n\n![Choosing the right database](https://cdn-images-1.medium.com/max/800/1*lQzxj0rx5WUBhinV2oaMsQ.jpeg)\n\n### Quick Notes on the NuoDB Architecture\n\n**NewSQL** systems are designed to operate in a distributed cluster of shared-nothing nodes, in which each node owns a subset of the data.\n\nAlthough it appears as a single, logical SQL database to the application, it has two-layers that retain strict transactional consistency. It can even be deployed across multiple availability zones (even on different clouds!) and is optimized for in-memory speeds, continuous availability, and adaptive scale-out that adjusts to the application needs.\n\n![NuoDB Architecture](/images/blog/nuodb-architecture.png) \n(***NuoDB Architecture: TEs (Top), SMs (Bottom)***)\n\nTransaction Engines (TEs): The TE layer is used for (ACID) SQL and caching, made up of in-memory process nodes that coordinate with each other and the SM layer.\n\nStorage Managers (SMs): The SM layer is used for storage and consists of process nodes that have both in-memory and on-disk storage components. SMs provide on-disk data durability guarantees, and multiple SMs can be used to increase data redundancy.\n\nBoth TE and SM node types can be easily added or removed to align with user needs.\n\n### Why Use OpenEBS for NuoDB?\n\nRunning applications on traditional virtualized systems or bare metal are different from running them on Kubernetes. NuoDB is a horizontally scalable database that offers scale up in performance on the fly from the underlying storage system. **Container Attached Storage** (CAS) solutions like OpenEBS are a perfect choice to run NuoDB on Kubernetes, both on-premise (such as OpenShift) and through the cloud because of the following benefits:\n\n- Local disks or cloud disks are managed seamlessly and in the same way by OpenEBS.\n- Persistent volumes (PVs) can be made available to NuoDB from a shared pool of disks. You can run multiple stateful databases out of the same physical disks present on the Kubernetes worker nodes, simplifying operations, increasing density, and decreasing costs.\n- Large size PVs can be provisioned for NuoDB. OpenEBS supports volumes up to petabytes in size.\n- You can increase the size of PVs as needed. With OpenEBS it is possible to start with small capacities and add disks as needed on the fly. Sometimes NuoDB instances are scaled up because of the capacity on the nodes. With OpenEBS persistent volumes, capacity can be thin provisioned, and disks can be added to OpenEBS on the fly without disruption.\n- NuoDB backups can be taken at the storage level and stored in S3. Using the backup/restore capabilities of OpenEBS, NuoDB instances can be moved across clouds or Kubernetes clusters seamlessly. There is no vendor lock-in problem when OpenEBS is used as the underlying storage.\n- Last, but not least — OpenEBS is 100% open source and in userspace.\n\n![production deployment of NuoDB](/images/blog/production-deployment-of-nuodb.png) (***Typical production deployment of NuoDB using OpenEBS.***)\n\nLet’s deploy our first NuoDB instance on our existing small, three-node OpenShift K8s cluster with OpenEBS.\n\n**Note:** NuoDB Community Edition supports only 1 SM and 3 TE configurations. To increase the number of SMs and TEs, the Enterprise edition should be used instead.\n\n### Requirements\n\n### Software\n\n- RHEL (master and node servers) 7.0 and above\n- [OpenShift](https://containerized.me/how-to-install-openebs-on-openshift/) 3.9 or above\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- [OpenEBS](https://github.com/openebs/openebs) v0.8.1 or above\n- NuoDB 3.4 Enterprise or Community Edition\n\n### Hardware\n\n- 3 nodes (bare metal or cloud)\n- 16 vCPUs\n- 32GB RAM\n\n### How to Install NuoDB\n\nIf you don’t have an OpenShift cluster, you can follow my previous blog [here](https://containerized.me/how-to-install-openebs-on-openshift/) to get both OpenShift and cloud-native storage OpenEBS up and running on Kubernetes.\n\nThe *NuoDB CE OpenShift* template is available with two options: using ephemeral storage or using persistent storage. With ephemeral storage, if the Admin Service pod or Storage Manager (SM) pod are stopped or deleted, the database will be removed with the pod as expected. That is where **OpenEBS** comes into play. When running NuoDB on OpenEBS, even if the pod is stopped or deleted, the DB state will be preserved and made available again on the automatic pod restart.\n\n### Disable Transparent Huge Pages (THP)\n\nAdditionally, on your OpenShift nodes run the command below to ensure Transparent Huge Pages (THP) is disabled. NuoDB will not run on a Linux machine with THP enabled.\n\n $ cat /sys/kernel/mm/transparent_hugepage/enabled\n\nIf this returns *|always| madvise never*, THP is enabled and you must disable it by adding the *transparent_hugepage=never* kernel parameter option to the grub2 configuration file. Append or change the “*transparent_hugepage=never*” kernel parameter on the *GRUB_CMDLINE_LINUX* option in the /etc/default/grub file similar to the below:\n\n $ vi /etc/default/grub GRUB_TIMEOUT=5 GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT=”console” GRUB_CMDLINE_LINUX=”nomodeset crashkernel=auto rd.lvm.lv=vg_os/lv_root rd.lvm.lv=vg_os/lv_swap rhgb quiet transparent_hugepage=never” GRUB_DISABLE_RECOVERY=”true”\n\n### Create a Project\n\nCreate a project named “nuodb” by clicking the OpenShift “Create Project” button. If desired, you can also add a template display name of “NuoDB on OpenEBS.”\n\n### Create an Image Pull Secret\n\nIf you want to use the official Red Hat Container Registry, then you need to create an image pull secret and import the image from the Red Hat Container Catalog (RHCC).\n\nTo create an image, use pull secret which allows the template to pull the NuoDB CE container image from the RHCC. From the OpenShift left bar menu, click “Resources,” then “Secrets,” and “Create Secret.”\n\nEnter these values:\n\n Secret Type = Image Secret\n Secret Name = pull-secret\n Authentication Type = Image RegistryCredentials \n Image Registry Server Address = registry.connect.redhat.com\n Username = (your RH login)\n Password = (your RH password)\n Email = (your email address)\n Link secret to a service account = (check this box)\n Service Account = Default\n\nThis should look similar to the screenshot below:\n\n![Create an Image Pull Secret](/images/blog/create-an-image-pull-secret.png)\n\nYou can find the RHCC instructions [here](https://access.redhat.com/containers/?tab=images&platform=openshift#/registry.connect.redhat.com/nuodb/nuodb-ce) if needed.\nAs an alternative, you can use the Docker registry images (not suggested for production).\n\n### Label the Nodes\n\nLabel at least three servers using the oc label command:\n\n $ oc label node nuodb.com/zone=nuodb\n\n### Configure cStor Pool\n\nAfter OpenEBS is installed, cStor pool must be configured. If cStor Pool is not already configured in your OpenEBS cluster, this can be done following the instructions [here](/docs/deprecated/spc-based-cstor#creating-cStor-storage-pools). A sample YAML named openebs-config.yaml used for configuring cStor Pool is provided in the Configuration details below. During cStor Pool creation, make sure that the maxPools parameter is set to >=3.\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get disks`\n #\n # `Disk` is a custom resource supported by OpenEBS with `node-disk-manager`\n # as the disk operator\n # Replace the following with actual disk CRs from your cluster `kubectl get disks`\n # Uncomment the below lines after updating the actual disk names.\n disks:\n diskList:\n # Replace the following with actual disk CRs from your cluster from `kubectl get disks`\n # - disk-184d99015253054c48c4aa3f17d137b1\n # - disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # - disk-806d3e77dd2e38f188fdaf9c46020bdc\n # - disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # - disk-bad1863742ce905e67978d082a721d61\n # - disk-d172a48ad8b0fb536b9984609b7ee653\n ---\n\n### Create the Storage Class\n\nYou must configure a StorageClass to provision a cStor volume on a given cStor pool. StorageClass is the interface through which most of the OpenEBS storage policies are defined. I will be using a StorageClass to consume the cStor Pool that is created using external disks attached on the Nodes. Since NuoDB is a statefulset application, it requires only a single storage replica. Therefore, the cStor volume replicaCount is =1. A sample YAML named openebs-nuodb.yaml used to consume cStor pool with cStoveVolume Replica count as 1 is provided in the configuration details below.\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-nuodb\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n - name: ReplicaCount\n value: \"1\" \n provisioner: openebs.io/provisioner-iscsi\n reclaimPolicy: Delete\n ---\n\n### Import the NuoDB CE Template\n\nImport the NuoDB CE template *ce-template-persistent.yaml* into OpenShift by navigating to the “Overview” Tab, clicking “Import YAML/JSON,” and running the import.\n\n![Import the NuoDB CE Template](/images/blog/import-the-nuodb-ce-template.png)\n(***Import YAML Template***)\n\nFollow the installation prompts and change the “SM persistent storage class name” and “Admin persistent storage class name” to *openebs-nuodb*.\n\n![Set Persistent Storage Class Name](/images/blog/set-persistent-storage-class-name.png) \n(***Set Persistent Storage Class Name***)\n\nClick “Create,” and shortly after you will see one pod each for the Administrative Service (Admin), Storage Manager (SM), Transaction Engine (TE) and Insights processes started under the *nuodb* namespace with volumes provided by OpenEBS.\n\n![nuoDB Successfully Deployed](/images/blog/nuodb-successfully-deployed.png) \n(***NuoDB Successfully Deployed***)\n\nIf you’d like to see or discuss anything not covered here, feel free to comment on my blog or contact me via Twitter @muratkarslioglu.\n\n---\n\n*Originally published at *[*containerized.me*](https://containerized.me/how-to-deploy-nuodb-on-kubernetes/)* on March 14, 2019.*\n","notHasFeatureImage":false,"slug":"how-to-deploy-nuodb-on-kubernetes-with-openebs"},{"id":57,"title":"Inspired by SCale17x — a short history of data storage","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"11-03-2019","tags":["DevOps","OpenEBS","Kubernetes","Open Source"],"excerpt":"Last weekend I had the pleasure of attending SCale 17x in Pasadena, just outside Los Angeles. Whether it was Robert Treat’s excellent talk on logical replication for PostgreSQL or refreshing my knowledge of Terraform","content":"\nLast weekend I had the pleasure of attending SCale 17x in Pasadena, just outside Los Angeles. Whether it was Robert Treat’s excellent talk on [logical replication for PostgreSQL](https://www.socallinuxexpo.org/scale/17x/presentations/postgres-logical-replication-lets-do-it-live) or refreshing my knowledge of Terraform thanks to the [dynamic and even funny instruction](https://www.socallinuxexpo.org/scale/17x/presentations/terraform-50-minutes) of [Rami Al-Ghanmi](https://twitter.com/alghanmi),  it was a great weekend filled with high-quality discussions. Kudos to Ilan and the rest of the all-volunteer organizing staff. Most importantly, it was a chance to run into friends, gather the MayaData/OpenEBS team together, support our CTO Jeffry as he told the story about [CAS (a great, deeper dive into the why and how of CAS)](https://www.socallinuxexpo.org/scale/17x/presentations/container-attached-storage-cas-openebs) and meet countless engineers, especially a bunch in the PostgreSQL, Percona, MariaDB, and MySQL communities.\n\nPerhaps my favorite moment was when I introduced myself during the DevOps LA taco night and the immediate reaction from a lead developer at a large analytics company was:\n\n“Oh holy #hit, OpenEBS and CAS — we need this stuff; working on it now as we move onto Kubernetes!”\n\nAs the conversation continued, a couple of questions emerged:\n\n1. How mature is it?\n2. Where did the idea come from?\n\nRegarding #1 — *maturity* — I think the growth of references on the community and our work at [www.openEBS.ci](http://www.openebs.ci/) to test every commit to master in public across many environments with many workloads both speak to the ongoing polishing and maturity of OpenEBS. I won’t dig too far into that subject, but I will say it is a lot more mature than many hacked together attempts at simply using a local PV. However, it is still <1.x.\n\nYou can read more about our efforts to mature OpenEBS and its docs and related management software in our founder’s recent blog post on the 0.8.1 release:\n\n[https://openebs.io/blog/openebs-releases-081-with-stability-fixes-and-improved-documentation](/blog/openebs-releases-081-with-stability-fixes-and-improved-documentation)\n\nMaturity and polishing is our top area of focus, and it is coming along quickly with the help of additional users and our dedicated, well-experienced engineering team. The next time you see them [on slack](https://openebs-community.slack.com/), please say thank you; the effort level by all involved is phenomenal.\n\nRegarding #2 — *where did the idea come from* — when trying to answer that question, I often roll the tape back a bit to see where storage has come from.\n\nWe put together a few slides that offer a light-hearted approach to answering this question.\n\n**The Era of Slow Disk Drives and Scale-up Architectures**\n\n![Slow Disk Drives and Scale-up Architectures](https://cdn-images-1.medium.com/max/800/0*6SPflCuTGC2rWXmH)\n\nSo, rolling the tape back to the 2000s, there were at least a few major differences as the slide expresses.\n\n1. At the bottom were slow disk drives; striping was required for adequate performance.\n2. Apps were not built for failure — users expected the storage to be bulletproof; on the other hand, once you got the App working, it wasn’t going anywhere, it was a monolith that could be tightly bound to the hardware and to the network attached storage.\n3. Along those lines, your apps were definitely not built to run on a public cloud service. They ran on a box, in your data center.\n\nBack then, it all sort of made sense. You needed software and systems that could combine many servers and disks into a highly resilient system to make sure your monolith continued to operate.\n\n**What has changed and how is that changing the data layer?**\n\n![Flash and NVMe extremely fast. Striping slows them down. DAS beats storage](https://cdn-images-1.medium.com/max/800/0*ZJ8CaKQdxEesL7eE)\n\nStorage was disrupted by flash. But now we are done with that, right? Well, not really…. Today’s scale-out storage systems almost always impose massive latency versus underlying, extremely fast NVMe connected non-volatile systems. As a result, and from a desire to avoid the complexity of running a distributed system like Cassandra on top of a distributed storage system (often with the distributed Kubernetes system in between), DAS has beaten all existing storage for many workloads.\n\nIn short, **the very reason that we started striping across nodes — to add resilience and performance to slow disks — no longer applies.**\n\nSo, what else?\n\n![75x increase in the number of workloads. 95% decrease in size & duration. Ephemeral](https://cdn-images-1.medium.com/max/800/0*VzhXdDLpxtERp1Sk)\n\nPerhaps most importantly, the workloads — the way DBs are built and run — has changed. They are of much shorter duration, typically much smaller, and they move around! This is not your father’s monolith!\n\n![You pay clouds for storage. Data gravity -> ^ spending, less agility. Lock-in](https://cdn-images-1.medium.com/max/800/0*Yuk0vAdoZe4qp24s)\n\nThen you run it on the cloud, which gives you and your teams incredible agility. However, you now introduce dependencies because each cloud’s storage differs. So, if you are not careful, before you know it you are locked-in. Even for some of our users listed here, ^^ their AWS bills are growing quickly. And, again, your teams are less agile because there has been a collective decision to grab some opaque service such as a stripped down and standardized version of open source MySQL or Redis from another company.\n\nTo sum things up, the world has changed in many ways! Therefore, storage has changed too, right? Well, not really. Storage is hard. It is far easier through marketing and open source influence peddling to get your 20-year-old project labelled “cloud native” than it is to actually start an entirely new storage system that runs anywhere.\n\n![Run anywhere with dynamic provisioning. Don't stripe across nodes. Reduce SPOF](https://cdn-images-1.medium.com/max/800/0*vNlv_UcOVV5Y456W)\n\nTo begin at the bottom of the stack, you not only have to accommodate the presence of extremely fast storage hardware, you also have to accommodate running *on clouds* themselves. How can you do this? And how do you do so in a way that does not itself introduce another single point of failure?\n\nThe approach we take at OpenEBS is to run our storage capabilities in a decoupled way so that the controller — that serves the actual workloads — is itself stateless and easily rescheduled. Also, it runs in the user space in containers so that it really can run anywhere. It then coordinates reads and writes to logically underlying data containers.\n\nWe also keep the data local and utilize intelligence in our control plane to leverage Kubernetes to optimize data placement of primary data stores and replicas. There is much more I could discuss here, but I’ll leave it at this high level for now.\n\n![Every workload has its own data storage. Every team as well. Quick & agile](https://cdn-images-1.medium.com/max/800/0*ZFFma1SdaG9GaiAS)\n\nSecondly, instead of having a one size fits all storage system that is limited in its ability to make trade-offs for each of the potentially thousands of workloads, each workload has its own storage system with OpenEBS. The specific needs of those workloads and of the “two pizza” teams that build and run the workloads are each served by their own OpenEBS. This limits the need for cross-functional team meetings in which members argue about how they want the storage to behave. They simply invoke their particular flavor via a Helm chart and, with the help of the right storage classes, away they go!\n\n![Run Dbs w/o any headache. Achieve data agility and cost savings. No lock-in. Unicorn](https://cdn-images-1.medium.com/max/800/0*w9bt1flsblASWqcG)\n\nBecause you can customize OpenEBS to run however you want it and it abstracts away the underlying differences from storage vendors or storage services, you are now less locked-in than before.\n\nPlus, with OpenEBS and MayaData we are providing per workload back-up and recovery and entire workload data migration as well. Common use cases include disaster recovery of stateful workloads, where you cannot, of course, have the workload itself doing all its own replication if the disaster you are protecting against is a catastrophic situation on that workload. Other common use cases include moving workloads from one cloud to another to save funds or possibly moving workloads closer to other cloud services.\n\n**What’s next?**\n\nNo look back to history would be complete without a quick look at the not-too-distant future as well.\n\nIn our case, we are adding additional performance tuning capabilities to the per workload container attached storage of OpenEBS. This will allow you to more easily tune components of a database, for example.\n\nAdditionally, we are working closely with a variety of workload providers to improve the ongoing testing and, yes, maturation of DBs + OpenEBS on various clouds and on premises deployments. We are increasingly getting help in building and operating [OpenEBS.ci](http://openebs.ci/) for example. The idea is to make it much easier to run your own databases without the loss of control and decreased agility from relying on cloud managed databases.\n\nHopefully you can see why I believe we have entered a golden age for building and running stateful applications. With the help of container attached storage like the solutions delivered by OpenEBS, you can run workloads anywhere in a completely automated way while providing a nearlt impossible level of per workload customization and performance. The net result is a boost in developer productivity and a reduction in cloud lock-in.\n\nWe are also inviting community members to write interesting blogs with hands-on content that OpenEBS and broader Kubernetes providers might find useful. So, if you have an idea for a blog post, such as doing blue/green with stateful workloads or a comparison between different Kubernetes platforms, please do get in touch. We would be happy to cross-publish on the OpenEBS blog and to otherwise collaborate with you.\n\nThanks again to the Southern California Linux community! It was an inspiring weekend.\n\nHere are the reference slides I mentioned (with a nice Monty Python GIF included as a bonus):\n\n[https://docs.google.com/presentation/d/11dNx7-HEqUg6ZeUAtaKAzfqfscmuDcXaKWbjxbM3us4/edit?usp=sharing](https://docs.google.com/presentation/d/11dNx7-HEqUg6ZeUAtaKAzfqfscmuDcXaKWbjxbM3us4/edit?usp=sharing)\n","notHasFeatureImage":false,"slug":"inspired-by-scale17x — a-short-history-of-data-storage"},{"id":58,"title":"Becoming a Kubernaut - Top Learnings in my journey of becoming a CKA","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"05-03-2019","tags":["Cka","Kubernauts","Kubernetes","MayaData","OpenEBS"],"excerpt":"I would like to tell you about my experience with the Certified Kubernetes Administrator (CKA) exam.","content":"\nHello, my fellow Kubernetes Developers,\n\nI would like to tell you about my experience with the Certified Kubernetes Administrator (CKA) exam.\n\nI was first exposed to Kubernetes when I started to work on OpenEBS, a Container Attached Storage solution.\n\nAfter working on the project for a period of time, I thought about taking the CKA certification exam. I took approximately one and a half months to practice and get my hands dirty. I completed the exam on the 28th of December 2018 and was excitedly able to pass the exam.\n\n![](https://lh6.googleusercontent.com/xjqXS1OHG4peRCNCbgIucCHlqEEgY4dzFr2vCot0pj0dRz075Xv7Ed2p9yBiqYMWLkvKOFLLIkXL-6lTtqIap981KBWebNZvfBn8SYmOstKa-SjdUruWyli92vsGatDvXt5Nveq2)\n\n## CKA is about making friends!\n\n![](https://lh6.googleusercontent.com/y-dr6deMc6gBAJs4UKCCdA_rmDTbRLyyUd5OLMj1u6_deX1JIDWVUJq-iQvBnJXKMrs3z5pqcT6FPEfaH8qJ80Gt0BdoQ5qOOEdDoy5ZI6KNfySYzr-dy52ojIb-U-W4ZiCXHJDR)\n\nWhen I was involved in developing the OpenEBS control plane, I was required to understand some basic concepts, specifically, the storage concepts of Kubernetes.\n\nI started to explore Kubernetes Docs which, in my opinion, is the greatest source for learning Kubernetes. I applied my newfound knowledge to the project. My in-house team members my company MayaData, namely Karthik, Amit, Vishnu, and Kiran just to name a few, groomed me and provided me with the right knowledge. From there, I started to develop more interest in the subject.\n\nIn mid-November 2018, I started to do an exhaustive walk through of Kubernetes docs and practiced it.\n\nObviously, there were a few things that did not make sense to me as Kubernetes covers a vast domain, e.g. Network, Linux, etc.\n\nThroughout my experience, I frequently used the Kubernetes Slack channel and asked questions. I also began to attend local meetups. From there, my journey in making friends and a Kubernetes Mastermind began. I met Suraj Narwade on one of the meetups, where he shared his experience on how he learned Kubernetes.\n\nI came to know about the Kubernaut Slack channel, space where you can join and ask about Kubernetes-related information, such as setting up CNI, performing workload operations, etc.\n\nThis journey was very exciting, and it became a journey to make new friends and, of course, this led to becoming a CKA.\n\n## About the Exam\n\n![](https://lh6.googleusercontent.com/8O1iFmqiXACiozyrrBeMHHX3GKf9HstipsrtEK7MnPyppP9kQVxxIL1aXogrCYs8fdOCYe1952aBSpqp2bokSxpMpvkn770m4wfMjuBlNEWeeETInd7cSX-l70GUq1o3QI3SonVq)\n\n- The duration of the exam is 3 hours.\n- There will be 6 Kubernetes clusters on which the questions are based.\n- There are 24 total questions.\n- You are allowed to open only Kubernetes Docs in the exam.\n\nVisit the following link to learn more about exam\n[https://www.cncf.io/certification/cka/](https://www.cncf.io/certification/cka/)\n\nRead the candidate handbook and exam tips PDF carefully. These can be found on the website link given above.\n\nTo learn more about the exam syllabus, visit the following link:\n\n[https://github.com/cncf/curriculum](https://github.com/cncf/curriculum)\n\n## Preparation and Tips\n\n![](https://lh3.googleusercontent.com/y5Y0lbm9j0gl9pMSf2pV9cAwHrLHzk2As5oReKk-gtz--IeZvCs4V1lPQeUKQImkpKQvoX3N7YGl-3OMMva5-vNzQpEYT6curv_PbRdDIZgMuKkGzx9wj4yD5CAc62xMU5YrOdid)\n\n- Go through the Kubernetes docs exhaustively, especially the concepts and task sections, so that when you see a question you can easily open the relevant topic in docs quickly.\n- Practice with Kubectl, e.g. create a deployment, service etc., without writing YAML.\n[https://kubernetes.io/docs/reference/kubectl/conventions/](https://kubernetes.io/docs/reference/kubectl/conventions/)\n\n- Avoid writing YAMLs as much as possible in the exam.\n- Don’t just read the docs, but put them into practice.\n- Know about systemctl commands.\n- Set up your own cluster from scratch using [Kubernetes Hard Way](https://github.com/kelseyhightower/kubernetes-the-hard-way). I applied a little trick here; first I became familiar with basic Kubernetes constructs, played with workloads(e.g. Deployment, daemon set, etc), services etc., on an already configured Kubernetes cluster. Then I did it the hard way.\n- Some folks also advise studying books and other learning materials, but I personally did not follow any books.\n\nI will paste some useful resource links that might be helpful in addition to the Kubernetes docs.\n\n[https://suraj.pro/post/journey-to-cka/](https://suraj.pro/post/journey-to-cka/)\n\n[https://github.com/walidshaari/Kubernetes-Certified-Administrator](https://github.com/walidshaari/Kubernetes-Certified-Administrator)\n\n[https://medium.com/@sonasingh46/static-pod-in-kubernetes-e3854507655f](https://medium.com/@sonasingh46/static-pod-in-kubernetes-e3854507655f)\n\n[https://medium.com/@sonasingh46/story-of-pod-manifest-to-running-8e4b38f074ec](https://medium.com/@sonasingh46/story-of-pod-manifest-to-running-8e4b38f074ec)\n\nDefinitely join the Google Kubernaut Slack channel.\nChannel id: [https://kubernauts-slack-join.herokuapp.com/](https://kubernauts-slack-join.herokuapp.com/)\n\nAll the best for your journey in making new friends and cracking the Certified Kubernetes Administrator (CKA) exam!\n","notHasFeatureImage":true,"slug":"becoming-a-kubernaut---top-learnings-in-my-journey-of-becoming-a-cka"},{"id":59,"title":"OpenEBS releases 0.8.1 with stability fixes and improved documentation","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"25-02-2019","tags":["CAS","Cloud Native Storage","Kubernetes","OpenEBS","Stateful Applications"],"excerpt":"OpenEBS 0.8.1 is released today. In spite of being a tagged as a minor release, it has significant stability improvements and few important features. ","content":"\nOpenEBS 0.8.1 is released today. In spite of being a tagged as a minor release, it has significant stability improvements and few important features. OpenEBS community is growing and we are receiving more feedback. Most of the issues fixed in this release are in response to the feedback received from the community. In this blog article, I will capture some important highlights of the 0.8.1 release, for more details on the actual bugs fixed, see release notes.\n\n**Upgrade recommendation:**\n\nWe are recommending users of OpenEBS with cStor with Fedora 29 please upgrade to 0.8.1 as it contains an important fix that can increase the stability of the environment including avoiding possible node reboots by addressing underlying iSCSI and Kubernetes bugs. We are available to help on the community so please do get in touch if we can be of assistance.\n\n---\n\n## Highlights of 0.8.1 release:\n\n### Stability fixes:\n![image](https://cdn-images-1.medium.com/max/800/1*7BRJmxRw7IKONv7KKAgnkg.png)\n#### iSCSI discovery bug leading to Kubernetes node reboots:\n\nSome community users reported that Kubernetes cluster is becoming unstable sometimes when OpenEBS is in use. We worked with community and debugged their issues to find the root cause. After deeper analysis, we found that Kubelet continues to increase in memory usage and finally resulting in that particular node going for a reboot, causing the cluster to becoming unstable. An [issue has been reported already in Kubernetes on Kubelet](https://github.com/kubernetes/kubernetes/issues/70890), and in our case, same situation is being triggered from a different set of components, namely open-iSCSI and Fedora Kernel.\n\nHere are more details:\n\n1. We discovered a [bug Ferora 29’s Kernel](https://bugzilla.redhat.com/show_bug.cgi?id=1679565) . The kernel does not respond to a socket read request if the request size is less than the watermark (in cStor target’s case, the water mark is set to 48) if the data becomes available after the read request is made. This works without issue if the data is already available when the request arrives at the kernel.\n2. Kubelet executes an iSCSI client command for discovery of the target. During this target discovery connection, the requested size to the kernel is 16 bytes and could result in the above mentioned situation if you are on Fedora 29. If the iSCSI client or initiator does not receive the response to the discovery request, it times out after 30 seconds and the connection is retried. After 5 retries, the iSCSI client hits a bug where it returns an infinite amount of data to Kubelet, which ends up reading all of the returned data. Kubelet then consumes the entire available memory and creates an out-of-memory situation, causing the node to reboot. We have created an [open issue at open-iscsi](https://github.com/open-iscsi/open-iscsi/issues/155) for this bug as well.\n\nIn summary, when using Fedora 29 and open-iscsi, Kubelet experiences an out-of-memory issue if the iSCSI target’s TCP watermark is set to any number greater than 16. It was set to 48 by cStor’s iSCSI target. In 0.8.1, we set the watermark level to 1 and avoided the above bugs or issues (in Fedora, open-iscsi and kubelet)\n\nAll users running cStor with Fedora 29 are advised to upgrade to 0.8.1 release.\n\n**Documentation improvements**\n\nOur docs also received significant improvements. Simplified instructions and getting-started steps are provided to new users, and more troubleshooting scenarios are covered for advanced users. We have also certified cStor 0.8.1 for use with many stateful applications like GitLab, ElasticSearch, Minio, and Prometheus with the latest Helm charts and user documentation updated accordingly.\n\n![New docs — Stateful Applications with OpenEBS](/images/blog/stateful-applications-in-openebs.png)\n\nUser guides for backup and restore and RWM support are also important parts of this new documentation.\n\n![Backup and Restore Applications using OpenEBS Persistent Volumes](/images/blog/backup-and-restores.png)\n\n**Summary:**\n\nAs stated before, we recommend all cStor users upgrade to 0.8.1. We always welcome your feedback, so if you would like to see a new feature or need to report a problem, write a GitHub issue at [https://github.com/openebs/openebs/issues](https://github.com/openebs/openebs/issues)\n\nJoin our slack community at [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.be744436714e40be6d0b15e325bdf0b3.1580126064003.1580126064003.1580126064003.1&__hssc=216392137.1.1580126064004&__hsfp=3765904294)\n\nEnjoy using the 0.8.1 code! :)\n\n. . .\n\n## Other Notable Additions\n\n**Liveness Check for cStorPool Pods**\n\nSometimes when using network disks such as cloud provider disks or iSCSI disks from local SAN, it is common to experience intermittent disconnection of locally mounted disks. In such scenarios, cStorPool could move into a suspended state and require a restart of the cStorPool instance pod on that node. A liveness check probe has been added in 0.8.1 to help fix this issue. In the event of network disks undergoing a network connection flip, the disks get disconnected, cStorPool instance becomes unresponsive, liveness check probe restarts the pod and disk connections are re-established.\n\nBecause cStor volumes are replicated into three cStorPool instances, losing a cStorPool instance temporarily because of a restart causes volumes to be resilvered from the other pools.\n\n**Ephemeral Disk Support**\n\nIn 0.8.0, cStor does not provide high availability of the data if volumes are set up for replication across local disks of cloud providers such as AWS instance stores or GCP local SSDs. The local SSDs on cloud providers do not retain the data when the VMs are rebooted because a completely new VM is provisioned with newly formatted local disks. This scenario was also discussed in [another blog post](/blog/introduction-to-openebs-cstor-pools-and-considerations-during-k8s-upgrades?__hstc=216392137.be744436714e40be6d0b15e325bdf0b3.1580126064003.1580126064003.1580126064003.1&__hssc=216392137.1.1580126064004&__hsfp=3765904294#3fb6) and in the [0.8.0 docs](/docs/deprecated/releases-0x#080---dec-07-2018).\n\nEphemeral disk support is also added for cStorPools in 0.8.1. Now, users can use AWS instance stores or any other ephemeral local SSDs for setting up a highly available Kubernetes native cloud storage across zones for their stateful needs. When a Kubernetes node is lost, a new node arises with the same label and the same number of disks. OpenEBS detects this new node and a new cStorPool instance is also created. After the pool instance is set up, cStor rebuilds the volume replicas into the new cStorPool instance from the other volume replicas. During this entire process, the volume replica quorum needs to be maintained. For more details on this feature, see the docs section.\n\n**Support to Extend cStorPools onto New Nodes**\n\ncStorPools expansion feature is another good addition in this release. In 0.8.0, the pools, once set up, could not be expanded to new nodes. In 0.8.1, support is added to expand StoragePoolClaim configuration to include more nodes. For example, if you have a StoragePoolClaim config with disks from 3 nodes, and if you are want to add a new cStorPool instance, you can add the disk CRs from the new node to spc-config.yaml and mark the max pools as 4. Once you apply the new spc-config.yaml, the pool configuration will be increased to 4 cStorPool instances.\n\n**cStor targetNodeSelector Policy**\n\nSetting node selectors to targets were limited exclusively to Jiva in 0.8.0. In this release, this policy is extended to cstor as well. With this feature, you can limit the cStorController pod scheduling to one or a set of nodes in the cluster, which gives flexibility to the administrator in managing the resources across nodes.\n\n**Jiva Replicas not Loading when Data is Fragmented**\n\nWhen Jiva volumes are deeply fragmented, the number of file extends increases to a large number. In such a situation, Jiva replicas were not able to load within a specified time limit, and re-connection was also resulting in the same issue. The 0.8.1 release fixes this issue and Jiva replicas load without issue, even when they are highly fragmented.\n","notHasFeatureImage":false,"slug":"openebs-releases-0.8.1-with-stability-fixes-and-improved-documentation"},{"id":60,"title":"HA vs. DR and HA² for your DB","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"21-02-2019","tags":["Cloud Storage","DevOps","Database Administration","OpenEBS","Kubernetes"],"excerpt":"This post is intended to share observations gathered from a couple of decades worth of experience in helping enterprises build resilient systems and presented based on feedback from Kubernetes and StackStorm users over the past 4-5 years.","content":"\n*Steps to prevent cascading failures.*\n\nThis post is intended to share observations gathered from a couple of decades worth of experience in helping enterprises build resilient systems and presented based on feedback from Kubernetes and StackStorm users over the past 4-5 years. As such, mileage may vary; I myself am still learning and offer this as a way for us all to learn together, so feedback is not just welcome, it is absolutely requested.\n\nHere is what I have seen too often, and is the basis of all the principles I share below. For all of us, as we build these systems of systems with more dependencies and dynamic relationships than any one human could possible fully understand, we want to make sure that we don’t generate opaque cascading failures.\n\nIn short, Don’t Injure Yourself (DIY)\n\nFor starters, don’t make your automation so intelligent that it knows how to kill nodes that are not responding without also understanding *why* that node might be moving slowly. As I have seen before, it could be that  your load peaked the day after Thanksgiving, and by pulling the slow nodes out of the queue you are simply shortening the time before all other nodes become overwhelmed. These are the things brownouts are made of.\n\nSo, how can you avoid being thrown off the end of your own automation treadmill?\n\nHere are a few hard-learned principles that I draw upon:\n\n1. **Shift down** — tackle failure as close as possible to the failure source to limit the risk of injuring yourself with cascading failures.\n2. **Build every layer** — and  every system,  so that it is built to fail.\n3. ***Build*** *every layer* — do not think you have DR when you have HA; and don’t think you have HA just because you have one workload that spans several clusters.\n4. **Infrastructure as code –** related to #3. Always; no black boxes. The desired state is in the repo. Yes, the control loop at the center of Kubernetes will make simple decisions to trade off workloads and behaviors to the best of its ability. Grok that, and insist on everything  by deploying this approach.\n5. **Simplicity**. Yes, this is the UNIX philosophy of do one thing and do it well. No, a human cannot grok the entire system in many cases, but they must be able to grok the behavior of each subsection.\n\nLet’s look at a few FAILURE examples. Below, I have included the entire Google slide deck that contains the discussed images for you to use and review. They are even animated in this slide deck!\n\nI’ll be working off this example of a Kubernetes environment. As you can see, we have two Kubernetes clusters, each of which have many pods. I show Pod A, B and C for each, but in reality, there would be many additional pods as well. We have some stateful workloads running across those clusters.\n\n![kubernetes environment example 1](/images/blog/kubernetes-environment-1.png)\n\nNow let’s assume you have similar environments on cloud A and data center B or similar. The idea is that we have multiple locations for similar multi-cluster Kubernetes.\n\n*It may be useful for these examples to keep in mind that the fundamental architecture of a Container Attached Storage solution like OpenEBS is to run within the clusters to provide storage to stateful workloads. Essentially, this is the piece that gives each workload its own file or block access (or s3 if you run Minio on top, for example) irrespective of the underlying cloud volumes, existing storage, disks or other hardware such as NVMe. *[*https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/*](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/)\n\n![kubernetes environment example 2](/images/blog/kubernetes-environment-2.png)\n\n**Failure Condition #1: Cloud Volume or Disk Dies**\n\nStarting at the bottom of that stack, what should you do when a disk dies or a cloud volume becomes unresponsive? This is shown in by the red “X” in the image below.\n\n![Cloud Volume or Disk Dies](/images/blog/cloud-volume-or-disk-dies.png)\n\nIf you have NoSQL or some sort of scale-out SQL running, then presumably you do not need to do anything, right? I mean, you can just degrade the pod and even the cluster and it should still work, correct? Technically, that is correct, but what is the cost? What happens to your system when, for example, a ring is being rebuilt or the cache is being repopulated? How much cross-node traffic is needed?\n\nPrinciple #1 is an adaption of shift left from DevOps — in this case, it is **shift down**. You fix the dying disk or the loss of the value at that level through a simple and easy to grok RAID. The RAID should be as simple as mirroring, so that you do not waste time rebuilding or resilvering disks. In other words, don’t use Raid 5 or 6 just to save space at the cost of complexity and longer rebuild times. This could easily degrade the POD and Cluster. You should absolutely build intelligence at the bottom of the stack to ensure you survive disk failure without creating a risk of cascading failure as a result of taxing the higher levels of the stack.\n\n**Failure Condition #2: A Lost POD that is Running a Stateful Workload**\n\nThis is where HA is put to use. However, this is not the same as disaster recovery, which involves recovering an entire environment from the loss of a data center. Rather HA is intended to keep the application functioning properly with no downtime, and it should respond seamlessly to small hiccups such as pod outages.\n\nKubernetes, of course, helps here by ensuring that multiple replicas of the actual workload can be kept alive. However, Kubernetes is not necessarily proactive when it comes to keeping stateful workloads alive. This is where a properly deployed DB layer will be useful and provide a layer of HA for the application. This guarantees that reads and writes of state for the application are never interrupted. Typically, this is done by implementing a quick failover (quicker than Kubernetes might notice, without help) to a destination for writes and multiple read replicas. However, architectures amongst the dozens of DBs vary. You can read more about this subject in a recent blog I wrote on running your own DBaaS that also touches on picking the right DB from the start:\n\n[https://openebs.io/blog/running-your-own-dbaas-based-on-your-preferred-dbs-kubernetes-operators-and-containerized-storage](/blog/running-your-own-dbaas-based-on-your-preferred-dbs-kubernetes-operators-and-containerized-storage)\n\nThe TL;DR from above includes consideration about which operators you want to use for your DBs. Of course, DBs and other stateful workloads need some imperative flow to recover from failures such as Do X (log failure), Do Y (direct writes and/or reads to the active replicas), Do Z (clean up the old node), etc.\n\nHaving configured your DB in a HA manner with the help of an operator, you are now faced with a decision. But then, do you want a storage layer replica as well?\n\n![storage layer replica](/images/blog/storage-layer-replica.png)\n\nTypically, the answer we see amongst our users is *no*. However, if you have a container attached storage solution like OpenEBS, you can decide on a per-workload basis. Perhaps you have one workload in which you are concerned that the DB itself will crash, so you want to be able to immediately reschedule and redeploy the DB onto a new node with the help of Kubernetes. This way you can have the data on the node even if the DB had not been running previously on that node. Maybe you want the write ahead log to be double resilient and you want the actual data files, but you decide it is not worth the trade off, or vice-versa. I’m not sure what to call this double redundancy — maybe we can call it **HA²**.\n\nThere are definite trade-offs in this case as you would now be asking the storage to do a write and/or synchronous replication to multiple locations before acknowledging the write. While this will become less expensive when NVMe is used as a protocol and a new class of persistent memory becomes prevalent, today it can drop your write IOPS by 30–40% depending on configurations of the DB and the storage itself among other factors. Please note that this is the kind of tuning we do with the help of partners like Datastax, NuoDB, and a number of local and global systems integrators. You can always ask around in the OpenEBS community and you will almost certainly find helpers.\n\n**Failure Condition #3: What Happened to my DB?**\n\nContinuing on the prior example, if you want to be prepared for a scenario in which the DB itself disappears, then you really should have storage or some other method for keeping a copy of the state of the database. As mentioned, if you do synchronous replication or simply have the writes written in multiple places through solutions like OpenEBS, then you get every write (consistency!) but at the cost of latency and additional loading on your network. On the other hand, if your storage solution can quickly take incremental snapshots of your storage and move those snapshots somewhere else then you can commit the writes quickly, then take/move the snapshots of that data. This eliminates the latency tax, but at the cost of your recovery point objective. So, you should be prepared to lose writes with this scenario.\n\n![latency tax](/images/blog/latency-tax.png)\n\nBy the way, storage is not the only way to add a layer of resilience to your application in cases of DB failure. We often see cases in which the DB is somewhat isolated from the actual traffic by a cache layer (hello Redis!). This cache layer can take all the writes for a period of time. In this case, having a DB deployed onto a new node with all but the most recent X writes will work just fine.\n\n**Failure Condition #4: Disaster Strikes!**\n\nIn this scenario, a hurricane shows up and floods your data center. One approach is to have every single byte written in a data center or cloud asynchronously replicated to a second cloud. Unfortunately this causes suboptimal performance in Cloud A and will not guarantee that Cloud B will function properly when the load switches over.\n\n![Disaster Strikes](/images/blog/disaster-strikes.png)\n\nSo what should you do? A common pattern we have noticed involves letting the individual teams responsible for each application determine what solution best fits their project. They must have an answer if they have an SLA that leads them towards true DR. However, in the end they can decide what they want to do. It could be that they are willing to work in a degraded mode for a period of time.\n\nHere is where the ability to snapshot and move entire applications may be a good solution. We have seen OpenEBS itself being used in this way quite often with the help of Heptio’s Velero (Formerly Ark) plug-in. We are now adding capabilities to make the management of the entire workflow simpler so that each team has the necessary control to decide their approach while the broader organization can see their approaches in aggregate. This is the so-called DMaaS subproject, due for initial release in OpenEBS 0.9.\n\nAnother pattern that I heard about occurred at Netflix many years ago, and is now known as chaos engineering. At MayaData, we employ a flavor of chaos engineering in our CI/CD end to end pipelines. For instance, we publish every commit to the OpenEBS master on [OpenEBS.ci](http://openebs.ci/) and subject them to chaos engineering under a variety of stateful workloads and across a number of environments from OpenShift to managed Kubernetes services like AKS. We have open-sourced all of this, enabling tooling for stateful workloads called Litmus.\n\nThe point here is to *apply chaos engineering to your HA and DR approaches so that you can actually see them work and fail*. Everything fails occasionally , including the systems that are intended to help you survive failure. What’s important is understanding the blast radius and determining how hard or easy it is to continue operation and recover with time?\n\nWhichever method you use, keep in mind that the granularity of control that containers, microservices and Kubernetes extended via container attached storage enables is of great benefit to you. It allows responsible teams to fully grok their component in this aspect of service resilience and minimizes the blast radius ideally down to the per team scope of control: typically the microservice itself. You can think of this approach as building in fire breaks to stop cascading failures.\n\nI hope this blog is of use to those of you wrestling with ways to ensure resilience while running real (stateful) workloads on Kubernetes. The good news is that OpenEBS and other open source projects in and around Kubernetes are quickly accumulating thousands or tens of thousands of production hours and there are many experts that frequent such channels and are often ready and willing to help.\n\nSome of this experience informs our docs in the OpenEBS community, including common patterns for workloads such as Minio, MySql, Cassandra, Elastic and many others: [https://openebs.io/docs/stateful-applications/mysql](/docs/stateful-applications/mysql)\n\nAs mentioned, you can also see these and other workloads on display as each commit to master for OpenEBS is tested against them. You can even choose to inject chaos into the testing of these workloads on OpenEBS as it is developed and matured: [https://openebs.ci/workload-dashboard](https://openebs.ci/workload-dashboard)\n\nFor example, here is the Percona dashboard that shows the ability to inject failures and to generate loads. And yes, that is live. Feel free to cause chaos on our dime!\n\n![Percona dashboard 1](/images/blog/percona-dashboard-1.png)\n\nLast but not least, I’ve tried to pull this blog together into a single picture. I’ve been guilty in the past of thinking that one solution is all that is needed, whether at the storage, the cache, the automation, the DB, or the message bus layer to ensure resilience and optimize trade-offs between resilience, cost, and performance. These days I fall back on the principles I discussed above.\n\n![Percona dashboard 2](/images/blog/percona-dashboard-2.png)\n\nAs you can see , I’m not a big fan of trying to add resilience to our systems by bolting on an opaque and monolithic storage layer. This can cause unknown behavior in the midst of unexplained change, i.e. cascading failures. Rather, we recommend injecting just enough storage into your environment by extending Kubernetes itself, where appropriate, with the help of projects like OpenEBS. If you do have a pre-existing storage layer, no problem. You can make it more resilient and extend far greater granular control of storage services by running a solution like OpenEBS on top.\n\nWhatever path you take, I really do think you will be best served by taking the necessary steps to avoid the risk of cascading failures. By solving issues as close to their occurrence as possible, allowing each team to choose their own approach, planning for failure of even the backup pieces, and by participating in transparent communities, you will be able to keep moving quickly without losing data.\n\nAs always, feedback is welcomed and encouraged.Slides: [https://docs.google.com/presentation/d/1HN25JftStju9-twyqjRowuZeQU0_WNbcQH7kO_v66aU/edit?usp=sharing](https://docs.google.com/presentation/d/1HN25JftStju9-twyqjRowuZeQU0_WNbcQH7kO_v66aU/edit?usp=sharing)\n","notHasFeatureImage":false,"slug":"ha-vs.-dr-and-ha²-for-your-db"},{"id":61,"title":"Running your own DBaaS based on your preferred DBs, Kubernetes operators and containerized storage","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"14-02-2019","tags":["DevOps","Kubernetes","OpenEBS","Stackstorm","Cassandra"],"excerpt":"This blog is intended to outline some things to think about when running your own DBaaS. While I recommend considering these, I also suggest that you should think long and hard about operations automation and, of course, underlying data resilience.","content":"\n**Why would you want to do that?**\n\n![workloads onto Kubernetes](/images/blog/workloads-onto-kubernetes.png)\nNearly every day I hear about someone looking to move more workloads onto Kubernetes. This makes sense as the above data from StackOverflow indicates that Kubernetes usage has skyrocketed.\n\nThis differs from 2017, when I was mostly asked  about how running Kubernetes via OpenShift, Rancher, Kubespray, or from the source compares to buying it by the drip from the Big 3.  Now at [MayaData](http://www.mayadata.io/) and in the [OpenEBS](http://www.openebs.io/) community, we are often asked for advice about running your own Redis or Cassandra or MongoDB. The comparison these users are making is the DB as a service offering from the Big 3, or even from the DB providers themselves.\n\nSo, the first question we need to ask  is *why*? Why would you want to take on the responsibility of doing this yourself? Without digressing too far, I’ll simply summarize the answer this way: “For the same reason we run Kubernetes.” Broadly speaking, users want to run Kubernetes either to go faster or to save money — or both. They want to do so while minimizing headaches and risks. Users want everything to be Kubernetes native and to simply work.\n\nOf course, when it comes to data,  security is a top priority as well. Many enterprises such as financial services and health care providers, or those concerned about AWS as a competitor choose to run Kubernetes on their own environments. This is in part due to concerns about sensitive data being viewed by the cloud provider or by a malicious attacker. And yes, we have seen many OpenEBS users use local encryption, which seems to validate that trend.\n\n**DBaaS Alternatives**\n\nWith each of the major cloud offerings, you can buy many flavors of DBaaS. Your choice will likely depend on your technical requirements and, of course, on your relationship with these cloud vendors. Generally speaking, you will want a high-quality solution that minimizes downtime and delivers exactly what your engineers need with minimum hassle.\n\nA common way to analyze the variety of DBs is to examine the CAP theorem and what your applications need. Typically, the rule of thumb is you pick two out of three. There are many, many blogs and discussions on picking a DB, but the idea is that there is an ever-growing number of DBs, and a subset of those are run as a service by tier-one clouds.\n\n![DBaaS Alternatives](/images/blog/dbaas-alternatives.png)\n\n**A quick aside on data base flavors:**\n\nThere are many available DB solutions, many of which have little in common other than their primary task of ordering data for faster storage or access for a particular use case. There are at least eight main types:\n\n- **Key value stores** — such as the ETCD inside of Kubernetes\n- **Column stores** — such as Amazon’s Redshift. These are generally most efficient for analytics. SAP Hana is a famous proprietary in memory DB often classified as a columnar DB\n- **Document stores** — such as MongoDB\n- **Graph databases** — such as Neo4J\n- **Row stores** — Although sometimes referred to as a columnar store, Cassandra actually belongs to this category. Generally, row oriented DBs are better at writes.\n- **In memory databases** — such as Redis (please note our [VP of Product](https://redislabs.com/redisconf19/) is speaking at the April 2–3 Redis conf on considerations on Kubernetes)\n- **“Synthetic”** — such as CockroachDB, NuoDB and TiDB. These are able to retain consistency while scaling horizontally reads and writes, so arguably stretching or even making moot the ACP trade off highlighted above\n- **Time series databases** such as Influx DB, which often runs under Prometheus, arguably the most common stateful workload on Kubernetes. This is the default monitoring solution; you can read more about using OpenEBS under Prometheus in this blog post from October 2018: [https://openebs.io/blog/using-openebs-as-the-tsdb-for-prometheus](/blog/using-openebs-as-the-tsdb-for-prometheus).\n\nRelatively old and well-known SQL solutions seem to be extremely prevalent for a variety of reasons , such as the decreasing sizes of some workloads as microservices take off. Architectures increasingly use message systems like Kafka along with service meshes to pass relevant information and  data back and forth,  as opposed to dropping everything into shared NoSQL platforms. This is likely one reason that we see an enormous amount of PostgreSQL and MySQL applications running on top of OpenEBS. There are some distributed SQL systems that scale out and yet retain SQL semantics. Notable examples include TiDB and Couchase, each of which have key value stores ordering underlying data structures, and NuoDB, which is based upon a distributed memory cache. In between are solutions such as Vitess that take MySQL (or similar) and manage sharding and some of the operational automation needed to scale MySQL horizontally.\n\nYou can get a subset of these varieties from the cloud ventures as services. So, why are users increasingly running their own DBaaS? In addition to the obvious points above, more control, including running the particular DB du jour, and less spending and fewer security concerns. Whether they are well founded or not,why else might users select to build and operate their own DBaaS?\n\n**Compose a Stack that Better Serves the DB**\n\n![Compose Stack](/images/blog/compose-stack.png)\n\nDBs are themselves composed of pieces of software that have different requirements. You also have choices as to which underlying storage engines you use with each database and how you configure these storage engines as well. As an example, depending on the version and underlying storage engine being utilized, MongoDB has a journal, various data directories, and a replication log. Cassandra has commit_logs, data_files, and saved_caches. Similarly, PostgreSQL has a WAL director and data directory. With today’s container attached storage systems and the storage class construct in Kubernetes, you can easily match each component of a DB to a well-suited and tuned underlying storage component.\n\nAs an aside, if you’d like to learn about the differences between  BTrees and LSM approaches, their underlying storage engines, and related concepts tying back to the underlying computer science, you would benefit from this talk given by Damian Gryski last year:\n\n[https://youtu.be/wxcCHvQeZ-U](https://youtu.be/wxcCHvQeZ-U)\n\nSo, while the permutations of DBs are nearly endless, with many examples of many types of DBs that can be configured differently, you can use all of them on a common storage layer across any underlying physical disk or cloud volume. This can be done with the help of a solution like OpenEBS that is typically automatically customized for each permutation.\n\nYou can read more about storage for each component of a DB in the documentation of the many flavors of DBs; for example, the MongoDB documentation states:\n\n![mongoDB documentation](/images/blog/mongodb-documentation.png)\n\nIt is probably worth also noting that to deal with disk or SSD failures, the MongoDB documentation also suggests the use of RAID 10.\n\n**Easily Integrate your Disparate Data Sources**\n\nWe have seen that DBs increasingly do only one thing, and do it very well. This microservices approach to DBs relies on systems like Kafka to perform cross DB and other data source integration and streaming. This is opposed to having one external storage system provide integration of all the data with all of the attendant blast radius and difficulties that could entail. Now, much of the data may end up in Snowflake or some on premise equivalent after an ETL process, of course. Ironically, integrating all of these data sources can sometimes be easier if these DBs are known to do one thing very well within your environment.\n\n**DevOps Autonomy**\n\nOne fundamental precept of DevOps is to have small teams pick their own tools. Having each team choose their own DB solution fits this pattern, whereas having one mega DBaaS solution often does not.\n\n**What’s the catch? Operations.**\n\nHere at MayaData, we are huge believers in the power of Kubernetes as a foundation for more automated operations. However, we also see quite a bit of churn or diversity in approaches to building and running stateful workloads.\n\nCurrently, it seems like every flavor of DB and stateful workload is coming out with at least one Kubernetes operator. And while that is great, it is also somewhat concerning. Having co-founded the leader in event driven automation, [StackStorm](https://stackstorm.com/), I’ve spent many years dealing with this day two operations stuff and, well, it is hard. In my opinion, we need to embrace workflow for some of this work, whether it is Argo that is now integrated into Kubeflow or StackStorm. This sports a new container friendly and extremely powerful workflow engine called Orquesta at its core. Incidentally, [Orquesta](https://github.com/StackStorm/orquesta) can be easily embedded into other projects, and I expect and hope to see this happening more as we continue in 2019. By workflow, I mean using an engine that makes explicit the common process events such as forks, joins, conditional pauses, for example, and does so in a way that allows humans to better grok and share these patterns.\n\nAlong these lines, back in October of 2015  I helped author a blog about how Netflix and others use StackStorm for Cassandra operations.\n\n[https://stackstorm.com/2015/10/02/tutorial-of-the-week-cassandra-auto-remediation/](https://stackstorm.com/2015/10/02/tutorial-of-the-week-cassandra-auto-remediation/)\n\nIf you are interested in workflows for operations, by the way, you might find interesting this overview of the space by my StackStorm co-founder Dmitri Zimine, which also discusses the offerings from the Big 3 cloud vendors, who are positioning their workflow engines as glue to tie together functions from their serverless offerings: [https://thenewstack.io/serverless-and-workflows-the-present-and-the-future/](https://thenewstack.io/serverless-and-workflows-the-present-and-the-future/)\n\nSo — where does storage fit into all of this?\n\nWhether or not you choose to use a generalizable approach to operators  or use one of the many one-offs emerging in the Kubernetes community,  you can approach a level of generalization through the use of Container Attached Storage. While storage and related capabilities will not completely care for your stateful workloads, it can provide some common services that every database needs, and thereby allow engineers to focus on those particular aspects of each DB that need their attention.\n\nI’ve assembled common requirements into a table, as we often get questions from the community (and investors, customers, and new team members) that suggest that the lines between the dozens of DBs fighting for attention and underlying containerized storage sometimes seems lost in all the hubbub. For example, did you know that storage engines for DBs are not “really storage”? Feel free to make comments below or even better, ask questions on StackOverflow or on our Slack community: [https://slack.openebs.io](https://slack.openebs.io/).\n\n![common requirements](/images/blog/requirements.png)\n\nWe’ve recently completed the testing and creation of some recommended storage classes with one of the largest NoSQL community projects. You can always see each commit to Master of OpenEBS being tested against a variety of common data based on [OpenEBS.ci](http://openebs.ci/) as well. Grab both the storage classes and the configuration for the systems doing this testing as well if you are interested.\n\n![workload dashboard](/images/blog/workload-dashboard.png)\n\n\n**Conclusion**\n\nThis blog was intended to outline some things to think about when running your own DBaaS. While I recommend considering these, I also suggest that you should think long and hard about operations automation and, of course, underlying data resilience. At MayaData, we are actively working on a number of DBaaS implementations for enterprises and are happy to share best practices via discussions on [Slack](https://slack.openebs.io/) and elsewhere. Please do get in touch. Together, we are enabling tremendous boosts in data agility, often with the help of DBaaS deployments. Whatever your permutation of DB or other stateful workloads, we are here to do our part in helping you to achieve true data agility.\n","notHasFeatureImage":false,"slug":"running-your-own-dbaas-based-on-your-preferred-dbs,-kubernetes-operators-and-containerized-storage"},{"id":62,"title":"Introduction to OpenEBS cStor Pools and considerations during K8s upgrades","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"31-01-2019","tags":["Cloud Native Storage","Kubernetes Upgrades","Persistent Storage","Persistent Volume","Kubernetes"],"excerpt":"Developers and DevOps administrators like Kubernetes for the way it has eased the tasks in their daily lives. It has changed the way applications are designed, developed and managed. ","content":"\n> My Kubernetes was just upgraded — what happened to my data?\n\n> And what if I’m using ephemeral disks and thin provisioning to save money?\n\nDevelopers and DevOps administrators like Kubernetes for the way it has eased the tasks in their daily lives. It has changed the way applications are designed, developed and managed. However, the churn of features and capabilities of Kubernetes continues to be high and frequent upgrades to the platform are not uncommon. Kubernetes development releases are coming out on a quarterly basis and production upgrades are common every 4 to 6 months. What is more — if you are running on a Kubernetes service you typically are not 100% in control of when and how those upgrades are performed. The infrastructure software that runs on Kubernetes needs to be cognitive of these upgrades so that application developers and Kubernetes administrators do not need to do any hand-holding during such upgrades.\n\nOpenEBS is an infrastructure software component for stateful applications running on Kubernetes as it provides persistent storage volumes and associated data management capabilities. OpenEBS as cloud native or `container attached` storage has seen increased adoption since the release (last year) of the cStor storage engine, probably thanks to its support for enterprise grade snapshots and clones as well as improvements to cross availability zone awareness, etc. OpenEBS is designed in such a way that the administrators manage the storage operations in a Kubernetes native way using kubectl commands.\n\nIn this blog, I would like to drive home three points:\n\n- Introduction to cStor pools and considerations for DevOps architects\n- How cStor volumes and pools work during Kubernetes upgrades?\n- Some dos and don’ts about cStor volumes and pools during Kubernetes upgrades\n\n## Introduction to cStor Pools\n\n![cStor Pools](/images/blog/cstor-pools.png)\n\n(***cStor pools provide thin provision and save up to 50% of your storage costs on clouds.***)\n\nA cStor-Pool (Shown in the above figure as Pool1) is a set of pool instances on each participating node, where a pool instance is formed over a set of disks. The cStor target places data replicas on each of the pool instances and synchronously replicates the volume data onto those replicas. The target manages the quorum logic and rebuilds the data among replicas when they become unavailable/available. For example, if a node disappears or is rescheduled, the workload will not notice as OpenEBS will continue serving the data.\n\nEach cStor pool instance provides enterprise-grade data protection features such as data consistency, copy-on-write snapshots, RAID resiliency of disks, self-healing of data, and thin provisioning of data. The pools can be scaled within the node by adding more disks on demand or as the used capacity crosses a preset threshold such as 80%. This thin provisioning capability of cStor Pool instances typically leads to a savings of about 50% when pools are setup over cloud provider disks such as EBS, GPD, etc. and helps to limit operational overhead.\n\n## Node NOT_READY and node reboot is common in Kubernetes, and OpenEBS storage requires volume quorum.\n\nAt the beginning of this post, I talked about Kubernetes platform upgrades that are happening more frequently than one would imagine. A common phenomenon during upgrades is the reboot of the worker nodes. CPU and memory pressures are another commonly observed reason for the worker node to go into the “NOT_READY” state. In many cases, a mass upgrade of a Kubernetes environment will cause all volume replicas hosted in the cStor pool instance to go offline, which will cause the cStor volumes to operate in an online state, but in degraded mode.\n\nThis scenario when more than one or all worker nodes are offline is when “Volume replica quorum” is especially important. A volume is considered to be online (or healthy) when the replicas are in quorum. In a three-replica setup, cStor target needs two volume replicas to be available and fully synced for the volume to be in quorum. cStor marks the volume as read-only when the replicas are not in quorum. This is needed to maintain data consistency and avoid potential data loss. Simultaneous or unmanaged node reboots can cause cStor volumes to lose quorum and cause temporary data unavailability. Again, this makes sense as the worker nodes themselves are out of commission.\n\n## Node Reboot Considerations\n\nDevOps architects and Kubernetes administrators need to be watchful of this situation when Kubernetes upgrades are occurring. Instigating a planned node reboot that leads some cStor volumes to lose quorum may not be desired. This is a specific case where container attached storage like OpenEBS has a far smaller “blast radius” than typical external scale out storage. In the case of OpenEBS, you will likely lose availability to your storage for a **particular workload** when you simultaneously reboot more than one node and cause volume replicas to lose quorum. However, in the case of shared scale out external storage, such a scenario will result in data unavailability for all of your workloads at once.\n\n`Nonetheless, when performing a Kubernetes reboot the question that is often asked is \"what is the blast radius right now\"? In other words, what volumes will lose quorum if a particular node is rebooted or lost?`\n\nYou can figure this out via kubectl.\n\n### Ephemeral Disks Scenario\n\nAs hinted at above, node reboots are a distinct worry while using hosted Kubernetes services such as GKE, EKS and AKS. The default behaviour of these solutions upon reboot is that the nodes come back with completely new disks, and the total data written onto the original disks is lost. Perhaps not coincidentally, this behaviour serves as a powerful incentive to pay the cloud provider more money for their persistent storage options, even if doing so may increase your lock-in. On the other hand, if OpenEBS pools are hosted on these local disks, the reboot of a node causes the volume replicas on that pool instance to be completely rebuilt on your other volume replicas. So, your workload continues to easily access data and OpenEBS deals with the clean up behind the scenes; this pattern is fairly common on the OpenEBS community and benefits users by allowing them to use more performant and less-expensive SSDs. Please note that this auto rebuild functionality is currently supported for Jiva Pools but NOT for cStor Pools, which is one reason we have not made cStor the default storage engine for OpenEBS. The ephemeral disk/pool support for rebuilding in cStor [is coming in 0.8.1 release](/docs/concepts/cstor?__hstc=216392137.b18f31a8a021a7fe3920ac461d353400.1580126597006.1580126597006.1580126597006.1&__hssc=216392137.1.1580126597007&__hsfp=3765904294#cstor-roadmap). Until then, it is advised NOT to use cStor pools on ephemeral disks. And yes, 0.8.1 is almost available. If you have been following along in the community, for example [OpenEBS.ci](http://openebs.ci/), you have seen test results increasingly turning green.\n\n## Conclusion:\n\nThe cStor pools feature in OpenEBS is helpful in managing the storage needs of cloud native applications in a Kubernetes native way. You may want to pay attention to planned node reboots to avoid temporary data unavailability during Kubernetes upgrades. Also, DO NOT use cStor pools on ephemeral disks until OpenEBS 0.8.1 release.\n\nThanks for reading!! If you are new to OpenEBS and need help getting started, engage in our wonderful slack community at [slack.openebs.io](http://slack.openebs.io/?__hstc=216392137.b18f31a8a021a7fe3920ac461d353400.1580126597006.1580126597006.1580126597006.1&__hssc=216392137.1.1580126597007&__hsfp=3765904294).\n","notHasFeatureImage":false,"slug":"introduction-to-openebs-cstor-pools-and-considerations-during-k8s-upgrades"},{"id":63,"title":"Performance tuning, CAS and OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"30-01-2019","tags":["Cassandra","Kubectl","Kubernetes","Mayaonline","OpenEBS","Uncategorized"],"excerpt":"Now and again — ok almost every day — we get a question on the community or from our commercial users about performance.","content":"\nNow and again — ok almost every day — we get a question on the community or from our commercial users about performance.\n\n![IOPS SCREENS FROM MAYAONLINE](/images/blog/iops-screen-from-mayaonline.png)\n\nAnd our typical response is “mileage may vary.” Sometimes we also mention that there are a bunch of tunables then talk to us for help. And by the way — we love to discuss performance requirements as it shows off how CAS is different and also we get to learn much more about how OpenEBS is being used.\n\nThis blog shares some high-level points and then shares info.\n\nFirst, it is worth noting that Container Attached Storage — where the workload is married to the storage controller — is by definition **`per workload storage`**. So when you compare OpenEBS to, for example, CEPH or some other pre-Kubernetes antediluvian scale-out storage system you risk apples to banana comparisons. While CAS will be constrained by the environment itself and how many resources OpenEBS utilizes from Kubernetes — it will not be directly impacted by the addition of more and more workloads. Those workloads are more or less an island as to themselves. By comparison, a system that checks with a metadata store to determine where to write and where to read the data and stripes data across nodes, for example, will have far greater performance impacts as more workloads are added.\n\nSecondly, as I mentioned on twitter the other day, sometimes the performance you are looking to optimize should be human as opposed to workload. With OpenEBS running natively in Kubernetes you can spin up storage in seconds on a per-workload basis — and manage it in a very familiar Kubernetes way. No external distributed system run on a foundation other than Kubernetes can be as simple to run and scale. You’ll need a team of experts to run that external distributed storage system — experts that arguably should be applied to running the distributed system you are betting upon — Kubernetes itself. And you’ll inevitably find that it takes weeks to understand any external distributed state store and years to understand how it will behave in the real world. By comparison, `if you know how to run Kubernetes, you already know how to run OpenEBS.` So, if you compare the human factor of storage performance — the humans and their ability to understand and run the storage software itself — there is no real comparison between OpenEBS and any other storage. (yes, I see my fellow members of the CAS category and I know what having a kernel module means for ongoing operations :)).\n\nAlright, enough with the high-altitude view! If you are still reading, I assume you want to know the details about tuning OpenEBS.\n\nThe good news is that you can tune cStor — which is our more enterprise-focused storage engine released in 2018 — based on the characteristics of your workload. We are working on adding these configurations directly to your storage classes as part of the forthcoming 0.9 release (we are on 0.8.1 currently) to support the infrastructure as a code pattern.\n\nHere are the two main tunings that can help improve performance:\n\n1. Scheduling the application and associated cStor storage pods\n2. Tuning the cStor volume based on the application IO pattern\n\n## Scheduling\n\n![Scheduling the Application and Associated cStor Storage Pods](/images/blog/scheduling-the-application-and-associated-ctor-storage-pods.png)\n\ncStor is the OpenEBS storage engine written in C. It is Kubernetes-native and follows the same constructs as other application pods, where affinity and anti-affinity rules can be specified to localize the access to the data and to distribute the storage pods to be resilient against node and zone failures. cStor uses synchronous replication behind the scenes to keep copies of your data, so the network latency between the cStor target and its replicas will have a performance impact that needs to be accounted for when designing the system for resiliency. Note that quorum policies are configurable, so you can specify if you want cStor to acknowledge writes when received or, more typically, when received and then confirmed by one or two replicas. Perhaps not surprisingly, performance is best when the application, the cStor target, and its replicas are co-located or connected via a fast network.\n\n## Tuning the cStor Volume\n\n![Tuning the cStor Volume Based on the Application IO Pattern](/images/blog/tuning-the-cstor-volume-based-on-the-application-io-pattern.png)\n\nThe two main tunings are LU, or “lun worker,” and queue depth, which is simply the depth of the queue used in IO for the OpenEBS target. By default, when you turn up OpenEBS and use the default configurations, OpenEBS runs in a lightweight manner. However, performance improves as you add more CPU to add more workers to LU and QD.\n\nIt is possible to modify the default settings via [`kubectl exec`](https://github.com/openebs/openebs/tree/master/k8s/demo/dbench) and if you would like to try these out, feel free to contact us through the OpenEBS Slack community. With 0.9, the tunable will be available as with other storage policies that OpenEBS currently supports via storage class and persistent volume claim annotations.\n\nYou can read more about these storage policies both for cStor and for the earlier Jiva based storage engine on our docs here:\n\n[https://openebs.io/docs](/docs?__hstc=216392137.181b23812f103703b848f80cc28e7104.1575964270497.1575964270497.1575964270497.1&__hssc=216392137.1.1575964270498&__hsfp=2230078507)\n\n![Types of Storage Policies for cStor](/images/blog/types-of-storage-policies-for-cstor.png)\n(***Types of Storage Policies for cStor***)\n\nKeep in mind that optimizing performance based on environmental and workload characteristics can be fairly complex. For example, performance often depends upon whether your workload is sequential or random and of course whether it is largely read or write. Many transactional DBs are sequential write-heavy, whereas analytic workloads are often sequential read-heavy and logging is typically more random writes. Also, the block sizes matter as does the characteristics of the underlying storage device (disks vs. SSDs for example). In the case of sequential workloads on cStor, your write performance may actually **degrade** with additional workers. In that case, you would want to decrease the workers to only one thread.\n\nAgain, remember that if you are on this path we are happy to help you and learn from your experience.\nWe have a variety of automated routines coming to assist in performance testing in OpenEBS 0.9 and later releases. There is a vast amount that can be done by closing the loop between configuration and actual performance as monitored — especially when combined with an active testing or chaos engineering solution such as Litmus. While there is some performance testing in Litmus, we and others in the community are adding workload tests such as TPS or transaction tests for various DBs that are frequently run on OpenEBS (thank you, Cassandra, NuoDB, MongoDB and other friends). We are also working on additional closed-loop optimization to build into OpenEBS operators. As always, we welcome your feedback and suggestions!\n\nSo, when might you want to make changes to tune your storage for performance? Typically, we see requests for these changes in environments in which storage is the primary performance bottleneck. As an example, we have a large IoT provider that is using cStor in thousands of remote nodes. This user knows the hardware underlying those nodes and their likely workloads. They felt that it was reasonable to maximize the resource usage of OpenEBS in order to expedite local on node performance of a random heavy workload.\n\nOn the other hand, if you are most interested in saving money and are running the workloads in clouds where CPU usage is relatively expensive, then you may prefer to run OpenEBS in a lightweight manner with as few workers as possible. Keep in mind that a common use case is running OpenEBS on persistent disks from the cloud providers, converting these more performant and less-expensive ephemeral resources into the foundation for resilient storage, thereby saving money while boosting performance.\n\nAs stated before, the mileage may vary from case to case, so feel free to contact us about your unique scenario. We are seeing many ways that necessary trade-offs between performance, resource usage, and workload and storage configurations can be better optimized on a per-workload basis. If you test many permutations of storage configurations against workloads configured in many ways, you can quickly find yourself in test scenarios that could take years to complete (gulp). It is far better to decompose the problem down to workloads so that the problem becomes more tractable. For example, you can configure MySql and the underlying OpenEBS storage differently depending on the use case, and even have systems in place that alter the configuration as needed. In other words, a fundamental advantage of the CAS architecture is that it enables a level of customization and refinement that shared everything storage approaches of the past made impossible.\n\nThanks for reading. Feedback and questions are welcome.\n","notHasFeatureImage":true,"slug":"performance-tuning,-cas-and-openebs"},{"id":64,"title":"Bare metal to production stateful workloads- OpenEBS and Rancher 2.x","author":"Dave Cook","author_info":"Building Cooby Cloud at Hetzner","date":"18-12-2018","tags":["OpenEBS","Stateful Workloads","Bare metal","Docker"],"excerpt":"Besides having a really cool name, Montel Intergalactic is also a cool company to work with. I met one of their engineers in an OpenEBS Slack forum and explained to him what I was up to","content":"\nBesides having a really cool name, [Montel Intergalactic](https://www.montel.fi/index.en.html) is also a cool company to work with. I met one of their engineers in an OpenEBS Slack forum and explained to him what I was up to. The next thing was that within a week we had everything setup at Hetzner. All Montel wanted in return was this humble blog post. A note to others in the space, THIS is how you build customer loyalty and a simple gesture like this can go a long way.\n\nNeedless to say when we decide on our next vendor these guys will be at the top of the list. In fact, they’ll probably be the only ones on that list (next to [OpenEBS](https://www.openebs.io/?__hstc=216392137.052f484bfa6105a863fd21af0f05de61.1579868893254.1579868893254.1579868893254.1&__hssc=216392137.1.1579868893255&__hsfp=3765904294) of course!).\n\nThis blog will outline the detailed steps we took to achieve our goal of setting up a bare-metal Kubernetes cluster at Hetzner from scratch using Rancher 2.x, Hetzner node driver, OpenEBS, Cert-Manager, Nginx Ingress loadbalancing and the deployment of the Tutem — Hello World app to demonstrate loadbalancing and Let’s Encrypt.\n\n**NOTE:** All nodes are created with Hetzner CX21 type servers. Our root domain is `cooby.tech` and an `A` record called `rancher.cooby.tech` points to the instance IP.\n\n### Initial Setup\n\nWe’ll be creating 4 Hetzner cluster servers:\n\n1. `front-1` nginx-ingress loadbalancing (type: worker)\n2. `control-1` control plane (type: etcd, control)\n3. `worker-1` first worker (type: worker)\n4. `worker-2` second worker (type: worker)\n\n- Spin up a Hetzner CX21 instance with a Ubuntu 18.04 image\n- Generate a keypair and make sure you can ssh into the instance\n- Install docker and docker-compose from your local machine:\n\n curl https://gist.githubusercontent.com/gridworkz/d78c290c4e6fd7753dc21bb50601745a/raw/625fc3b7ddb7a654fe379f67d0bb9c6ac5b9413a/InstallDockerCompose | ssh -i ~/.ssh/ root@rancher.cooby.tech\n “/bin/bash -s”\n\n- Login to the instance and add a docker-compose.yml file to run Rancher:\n\n ![Login to instance](/images/blog/login-to-instance.png)\n\n- Run:\n\n docker-compose up -d\n\n- `Install Hetzner node-driver from:` [https://mxschmitt.github.io](https://mxschmitt.github.io/ui-driver-hetzner/)\n- `Create a node-template for frontend`\n- `Add label loadbalancer=true`\n- `Remove iscsi driver with cloud-init:`\n\n![Remove iscsi driver](/images/blog/remove-iscsi-driver.png)\n- Create a node template for workers using the same cloud-config\n\n## Application Configuration\n\nSchedule pods so that they are not running on a node with tag **loadbalancer**. This is not a hard requirement, but in our example cluster it really makes sense.\n\n![application config](/images/blog/application-config.png)\n\n## Adding Nginx-Ingress\n\nDNS scheme is configured like this:\n\n- `.cooby.tech` A <`loadbalancer machines`>\n\nWhich enables you to use:\n\n- **Example 1.** `cooby.tech` and to use nginx ingress to route it to the example1 workload. Also in the DNS server, you can do, for example, cooby.in.montel.care CNAME montel.cooby.tech and again use our nginx-loadbalancer to route cooby.in.montel.care to your desired workload.\n\n**Pro Tip:** Once an Ingress service is created it defaults to port 42. Remember to edit the services’ YAML and change it to the port your application requires. You will save yourself hours of confusion.\n\n## Let’s Encrypt Issuer\n\nThe simplest way to add Let’s Encrypt in Rancher is to install it via catalog apps.\n\nAfter that every ingress that contains the following annotations:\n\n![ingress](/images/blog/ingress.png)\n\nWill be automatically handled by Let’s Encrypt.\n\nThe first time you add an ingress you have to give the certificate a name by editing YAML and adding only the underlined parts below. Other ingresses can be added within the Rancher UI.\n\n![rancher-ui](/images/blog/rancher-ui.png)\n\n## Adding OpenEBS\n\n- Login to each node and run: *apt-get install open-iscsi*\n- Confirm that open-iscsi is active: *systemctl status open-iscsi*\n- Navigate to global -> catalog in Rancher UI and enable Helm/Stable\n- Select OpenEBS and install it via Helm Charts\n\nThis will install base OpenEBS complete with storage classes. The next step is to create a persistent volume / persistent volume claim. This can be easily accomplished by running the following YAML:\n\n![pvc](/images/blog/pvc.png)\n\nThis will create a persistent volume claim named demo-vol1-claim with 10 GB of space in the storageClass: openebs-jiva-default. This storageClassName can then be passed as a parameter to various applications that require persistent and scale-able storage.\n\n## Conclusion\n\nIn this blog, we set out to build a simple Kubernetes cluster on Hetzner server instances. The goal was to put together an introduction to building Kubernetes from scratch by using some of the great tools that are out there that make this process simpler. This should act as a good foundation for any development or POC system. In fact, we use this setup over at [Cooby Cloud](http://cooby.io/). With the goodwill gesture from [Montel Intergalactic](https://www.montel.fi/index.en.html), I was able to get this up and running within a week. Hats off to them and the great folks on the [OpenEBS](https://www.openebs.io/?__hstc=216392137.052f484bfa6105a863fd21af0f05de61.1579868893254.1579868893254.1579868893254.1&__hssc=216392137.1.1579868893255&__hsfp=3765904294) Slack channel.\n\nNext up in this series I’ll demonstrate how to install Postgres and Odoo on this design utilizing some of the wicked tools over at [XOE Solutions](https://xoe.solutions/).\n\n`Originally published at `[coobycloud.blogspot.com](https://coobycloud.blogspot.com/)` on December 19, 2018.`\n","notHasFeatureImage":true,"slug":"bare-metal-to-production-stateful-workloads--openebs-and-rancher-2.x"},{"id":65,"title":"OpenEBS Snapshots using Kubectl","author":"Prateek Pandey","author_info":"Contributor and Maintainer @OpenEBS. Software Developer at @mayadata_inc. Open Source Enthusiast","date":"14-12-2018","tags":["Clone","Kubernetes","Snapshot","Storage"],"excerpt":"Kubernetes has a very “pluggable” method for adding your own logic in the form of a controller using CustomResourceDefinition(CRD).","content":"\nIn Kubernetes 1.8, many of the changes involve storage. For example, volume snapshotting API has been released as a ‘prototype’ level. It is external to core Kubernetes API’s, and you can find the project under the snapshot subdirectory of the [kubernetes-incubator/external-storage](https://github.com/kubernetes-incubator/external-storage) repository. There is, however, an ongoing proposal to add them as a core Kubernetes APIs here. For a detailed explanation of the implementation of volume snapshotting, you can read the design proposal here. The prototype currently supports GCE PD, AWS EBS, OpenStack Cinder, Gluster, Kubernetes hostPath, and OpenEBS volumes. It is important to note that aside from hostPath volumes, the logic for snapshotting a volume is implemented by cloud providers and core Kubernetes storage providers. The purpose of volume snapshotting in Kubernetes is to provide a common API for negotiating with different cloud providers in order to take snapshots and restore it as new persistent volume.\n\nKubernetes has a very `pluggable` method for adding your own logic in the form of a controller using [CustomResourceDefinition](https://kubernetes.io/docs/concepts/api-extension/custom-resources/#customresourcedefinitions)(CRD). VolumeSnapshot and VolumeSnapshotData are the two new CustomResources, and will be registered with the Kubernetes API server. This [user guide](https://github.com/kubernetes-incubator/external-storage/blob/master/snapshot/doc/user-guide.md#lifecycle-of-a-volume-snapshot-and-volume-snapshot-data) provides an overview of the lifecycle of these two resources. **`Snapshot-controller`** will create a CRD for each of these two CustomResources when it starts and will also watch for VolumeSnapshot resources. It will take snapshots of the volumes based on the referred snapshot plugin. **`Snapshot-provisioner`** will be used to restore a snapshot as a new persistent volume via dynamic provisioning.\n\nThe OpenEBS operator will deploy each **`Snapshot-controller`** and **`Snapshot-provisioner`** container inside the single pod called snapshot-controller.\n\n apiVersion: v1\n kind: ServiceAccount\n metadata:\n name: snapshot-controller-runner\n namespace: default\n ---\n apiVersion: rbac.authorization.k8s.io/v1beta1\n kind: ClusterRole\n metadata:\n name: snapshot-controller-role\n namespace: default\n rules:\n - apiGroups: [\"\"]\n resources: [\"pods\"]\n verbs: [\"get\", \"list\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumes\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumeclaims\"]\n verbs: [\"get\", \"list\", \"watch\", \"update\"]\n - apiGroups: [\"storage.k8s.io\"]\n resources: [\"storageclasses\"]\n verbs: [\"get\", \"list\", \"watch\"]\n - apiGroups: [\"\"]\n resources: [\"events\"]\n verbs: [\"list\", \"watch\", \"create\", \"update\", \"patch\"]\n - apiGroups: [\"apiextensions.k8s.io\"]\n resources: [\"customresourcedefinitions\"]\n verbs: [\"create\", \"list\", \"watch\", \"delete\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshots\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"volumesnapshot.external-storage.k8s.io\"]\n resources: [\"volumesnapshotdatas\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"services\"]\n verbs: [\"get\"]\n ---\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1beta1\n metadata:\n name: snapshot-controller\n namespace: default\n roleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: snapshot-controller-role\n subjects:\n - kind: ServiceAccount\n name: snapshot-controller-runner\n namespace: default\n ---\n kind: Deployment\n apiVersion: extensions/v1beta1\n metadata:\n name: snapshot-controller\n namespace: default\n spec:\n replicas: 1\n strategy:\n type: Recreate\n template:\n metadata:\n labels:\n app: snapshot-controller\n spec:\n serviceAccountName: snapshot-controller-runner\n containers:\n - name: snapshot-controller\n image: openebs/snapshot-controller:ci\n imagePullPolicy: Always\n - name: snapshot-provisioner\n image: openebs/snapshot-provisioner:ci\n imagePullPolicy: Always\n\nOnce **`Snapshot-controller`** is running, you will be able to see the created CustomResourceDefinitions(CRD).\n\n $ kubectl get crd\n NAME AGE\n volumesnapshotdatas.volumesnapshot.external-storage.k8s.io 1m\n volumesnapshots.volumesnapshot.external-storage.k8s.io 1m\n\n### Create Snapshot:\n\nTo create a snapshot, let’s now create the `PersistentVolumeClaim` to be snapshotted.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol1-claim\n namespace: default\n spec:\n storageClassName: openebs-standard\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nNote that this uses the openebs-standard StorageClass, which will dynamically provision an OpenEBS PersistentVolume. Let’s now create a Pod that will create data in the volume. We will take a snapshot of this data and restore it later. For example, in a busy-box application Pod, create date.txt and hostname.txt files in a mounted openEBS volume.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: busybox\n namespace: default\n spec:\n containers:\n — command:\n — sh\n — -c\n — ‘date > /mnt/store1/date.txt; hostname >> /mnt/store1/hostname.txt; tail -f /dev/null;’\n image: busybox\n imagePullPolicy: Always\n name: busybox\n volumeMounts:\n — mountPath: /mnt/store1\n name: demo-vol1\n volumes:\n — name: demo-vol1\n persistentVolumeClaim:\n claimName: demo-vol1-claim\n\nOnce the busybox pod is in a running state, we are ready to take a snapshot. After we create the VolumeSnapshot resource below, the snapshot-controller will attempt to create the snapshot by interacting with the OpenEBS snapshot APIs. If successful, the VolumeSnapshot resource is bound to a corresponding VolumeSnapshotData resource. To create the snapshot, we need to reference the PersistentVolumeClaim name in the snapshot spec that references the data we want to capture.\n\n $ cat snapshot.yaml\n apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshot\n metadata:\n name: snapshot-demo\n namespace: default\n spec:\n persistentVolumeClaimName: demo-vol1-claim\n \n $ kubectl create -f snapshot.yaml\n volumesnapshot \"snapshot-demo\" created\n $ kubectl get volumesnapshot \n NAME AGE \n snapshot-demo 18s\n\nThe conditions listed towards the bottom of the output below show that our snapshot was indeed created successfully. We can also check the snapshot controller’s logs to verify this.\n\n $ kubectl get volumesnapshot -o yaml\n apiVersion: v1\n items:\n - apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshot\n metadata:\n clusterName: \"\"\n creationTimestamp: 2018-01-24T06:58:38Z\n generation: 0\n labels:\n SnapshotMetadata-PVName: pvc-f1c1fdf2-00d2-11e8-acdc-54e1ad0c1ccc\n SnapshotMetadata-Timestamp: \"1516777187974315350\"\n name: snapshot-demo\n namespace: default\n resourceVersion: \"1345\"\n selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/fastfurious\n uid: 014ec851-00d4-11e8-acdc-54e1ad0c1ccc\n spec:\n persistentVolumeClaimName: demo-vol1-claim\n snapshotDataName: k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n status:\n conditions:\n - lastTransitionTime: 2018-01-24T06:59:48Z\n message: Snapshot created successfully\n reason: \"\"\n status: \"True\"\n type: Ready\n creationTimestamp: null\n\nWe can now look at the corresponding VolumeSnapshotData resource that was created. Notice the reference to the openebsVolume snapshot under spec, which also references the VolumeSnapshot resource we created and the PersistentVolume that from which the snapshot was taken. This PersistentVolume was dynamically provisioned by openebs-provisioner when we created our demo-vol1-claim PersistentVolumeClaim earlier.\n\n kubectl get volumesnapshotdata -o yaml\n apiVersion: volumesnapshot.external-storage.k8s.io/v1\n kind: VolumeSnapshotData\n metadata:\n clusterName: \"\"\n creationTimestamp: 2018-01-24T06:59:48Z\n name: k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n namespace: \"\"\n resourceVersion: \"1344\"\n selfLink: /apis/volumesnapshot.external-storage.k8s.io/v1/k8s-volume-snapshot-2a788036-00d4-11e8-9aa2-54e1ad0c1ccc\n uid: 2a789f5a-00d4-11e8-acdc-54e1ad0c1ccc\n spec:\n openebsVolume:\n snapshotId: pvc-f1c1fdf2-00d2-11e8-acdc 54e1ad0c1ccc_1516777187978793304\n persistentVolumeRef:\n kind: PersistentVolume\n name: pvc-f1c1fdf2-00d2-11e8-acdc-54e1ad0c1ccc\n volumeSnapshotRef:\n kind: VolumeSnapshot\n name: default/snapshot-demo\n status:\n conditions:\n - lastTransitionTime: null\n message: Snapshot created successfully\n reason: \"\"\n status: \"True\"\n type: Ready\n creationTimestamp: null\n kind: List\n metadata:\n resourceVersion: \"\"\n selfLink: \"\"\n\n### Restore Snapshot:\n\nNow that we have created a snapshot, we can restore it to a new PVC. To do this, we need to create a special StorageClass implemented by snapshot-provisioner. We will then create a PersistentVolumeClaim referencing this StorageClass to dynamically provision a new PersistentVolume. An annotation on the PersistentVolumeClaim will communicate to **`Snapshot-provisioner`** where to find the information it needs to deal with the OpenEBS API server to restore the snapshot. The StorageClass can be defined according to the code below.  Here, the provisioner field in the spec defines which provisioner should be used and what parameters should be passed to that provisioner when dynamic provisioning is invoked.\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: snapshot-promoter\n provisioner: volumesnapshot.external-storage.k8s.io/snapshot-promoter\n\nWe can now create a PersistentVolumeClaim that will use the StorageClass to dynamically provision a PersistentVolume that contains the info of our snapshot. The annotation field snapshot.alpha.kubernetes.io/snapshot refers to the VolumeSnapshot object. Snapshot-provisioner will use this resource to obtain the information it needs to perform the snapshot restore. After this, snapshot-provisioner will provision a PersistentVolume containing the contents of the snapshot-demo snapshot\n\n apiVersion: v1\n kind: PersistentVolumeClaim\n metadata:\n name: demo-snap-vol-claim\n annotations:\n snapshot.alpha.kubernetes.io/snapshot: snapshot-demo\n spec:\n storageClassName: snapshot-promoter\n accessModes: ReadWriteOnce\n resources:\n requests:\n storage: 5Gi\n\nWe can check the state of demo-snap-vol-claim to see if it is Bound or not. We can also view the snapshot-provisioner logs to verify that the snapshot was restored successfully.\n\n $ kubectl logs snapshot-controller-66f7c56c4-ptjmb-c snapshot-provisioner \n ... \n ... \n I1104 11:59:10.563990 1 controller.go:813] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" for claim \"default/demo-snap-vol-claim\" created \n I1104 11:59:10.987620 1 controller.go:830] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" for claim \"default/demo-snap-vol-claim\" saved \n I1104 11:59:10.987740 1 controller.go:866] volume \"pvc-8eed96e4-c157-11e7-8910-42010a840164\" provisioned for claim \"default/demo-snap-vol-claim\"\n\nNow, let’s mount the `demo-snap-vol-claim` PersistentVolumeClaim onto a busybox-snapshot Pod to check whether the snapshot was restored properly. After the busybox-snapshot pod is in a running state, we can check the integrity of the files that were created before taking the snapshot.\n\n apiVersion: v1\n kind: Pod\n metadata:\n name: busybox-snapshot\n namespace: default\n spec:\n containers:\n - command:\n - sh\n - -c\n - 'tail -f /dev/null'\n image: busybox\n imagePullPolicy: Always\n name: busybox\n volumeMounts:\n - mountPath: /mnt/store2\n name: demo-snap-vol\n volumes:\n - name: demo-snap-vol\n persistentVolumeClaim:\n claimName: demo-snap-vol-claim\n\n**Clean-Up:** \n\nWe can delete the VolumeSnapshot resource, which will also delete the corresponding VolumeSnapshotData resource from K8s. This will not affect any PersistentVolumeClaims or PersistentVolumes we have already provisioned using the snapshot. Conversely, deleting any PersistentVolumeClaims or PersistentVolumes that have been used to create the snapshot or have been provisioned using a snapshot will not delete the snapshot from the OpenEBS backend. As such, we must delete them manually.\n","notHasFeatureImage":false,"slug":"openebs-snapshots-using-kubectl"},{"id":66,"title":"OpenEBS 0.8 release allows you to Snapshot and Clone cStor Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"07-12-2018","tags":["Cloud Native","Kubernetes","OpenEBS","Storage","DevOps"],"excerpt":"cStor as a new Storage Engine and Node Disk Manager (NDM) that were introduced in the previous release have proven to be quite popular with DevOps user community.","content":"\ncStor as a new Storage Engine and Node Disk Manager (NDM) that were introduced in the previous release have proven to be quite popular with DevOps user community.\n\nThe following picture shows the locations where OpenEBS is deployed. `(More on this image later in the blog.)`\n\nIMO, this rise in adoption/installations can be attributed:\n\n- Enterprises are getting serious about Agility promised by the Cloud Native Architectures, where Kubernetes is an essential piece.\n- Kubernetes mainly focuses on the workflow of attaching Storage to Applications, but not managing the Storage itself. The Agility that Cloud Native Architectures bring are somewhat lost when a dependency is created on one specialized team ( Storage IT Team) which essentially slows down the Application Delivery Process.\n- Enterprises want the same experience with their Kubernetes Clusters whether they opt for Managed Kubernetes Clusters or have their own Kubernetes Clusters either On Premise or in Public/Private Cloud.\n\n`cStor Storage Engine` — being developed in the user space doesn’t require any special Kernel taints and just works on `any platform` of user’s choice — whether On Premise or On Cloud.\n\n`NDM` — addresses another missing aspect from Kubernetes about handling various types of Disks (or Storage Devices) connected to the Kubernetes Nodes and managing them, the Kubernetes way!\n\nAs one CTO evaluating OpenEBS, put it:\n\n> `I love OpenEBS Architecture, it helps remove the layers from the application delivery process. There is no need for a dedicated storage administrator or team. I can easily scale up and down this solution.`\n\nIn the spirit of Open Source Transparency, the CTO reached out to us to share his admiration for OpenEBS and express his concern over the performance of OpenEBS running completely in Userspace. The fact that none of the other Storage Providers do this and when push comes to shove,the CTO office needs to convince the DevOps Teams to pick OpenEBS over traditional/non-cloud native Storage Systems. DevOps Teams are aware of exactly where those legacy solutions break due to their distributed nature, and they also hold a long sustained perception of Kernel being faster than Userspace.\n\nWe knew performance is going to surface as a concern sooner or later. However, we have also found that users who have evaluated OpenEBS against other solutions still pick OpenEBS for its ease of use even if they incur a slight overhead.\n\nHaving said that, performance is still a real concern when it comes to Enterprises that run heavy computational jobs, and we know it. We have been working on resolving this performance piece for quite some time now and will very soon announce the details of what we have come up with. It suffices to say that “Performance with OpenEBS will not be a concern” in the near future.\n\nBut I digress.\n\nSpeaking of 0.8 release, some interesting features that I like are:\n\n- Support for cStor Snapshot and Clones\n- Support for managing cStor Volumes via kubectl\n\nAnd there are several other enhancements and bug fixes that went in along with a huge number of PRs from external contributors to improve code readability and unit tests. A detailed release notes can be found [here](https://github.com/openebs/openebs/releases/tag/0.8).\n\nThe OpenEBS community has grokked items like: Backup/Restore of OpenEBS Volumes to S3 compatible storage using Heptio ARK, Getting around the quirks of RKE to get iSCSI working.\n\nThe E2e team has also been very busy with setting up [https://openebs.ci/](https://openebs.ci/) that is powered by OpenEBS Litmus project with Chaos injection tools etc., Sharing the details about this project warrants for another blog!\n\n—\n\nLet me stick in this blog primarily to the cStor 0.8 features:\n\n**Snapshots and Clones.** While this feature is primarily viewed from data protection and recovery aspect, it has been found to make sense for many other use cases like:\n\n- CI / CD Pipelines for Stateful Workloads. When using OpenEBS as the underlying storage, the CI tools can make use of the Snapshot functionality to freeze the state of the application in case of build failure (or success). Developers can then launch their application with the frozen state of data from the CI pipeline for further debugging or detailed analysis. Saves a ton of time from having to reproduce the issues.\n- Big Data. The Analysis Workloads typically involve downloading large amounts of data from external sources into the Kubernetes volumes (a process called — warm-up) before running the intended computation jobs. And oftentimes the data downloaded doesn’t change very frequently. With cStor volumes, the data can be downloaded once and snapshotted. The computations jobs can use the cloned volumes. Save on network bandwidth and faster processing of the data. All this without losing the granularity of isolating the different workloads from each other.\n- Improving the Onboarding experience of users. SaaS Platforms such as — exam or certification portals or development IDEs require a set of prerequisite packages to be downloaded and installed. Snapshots can help in making sure prerequisite activities are performed outside of the Onboarding workflow. When a user requests for a service, a volume can be rapidly provided by cloning from snapshots. User experience matters! 5 mins to get your IDE or exam to being are not acceptable anymore.\n\ncStor Snapshot and Clone functionality are triggered via **kubectl** itself. cStor Snapshots and Clones are reference based and are highly optimized for lowering the capacity required for maintaining large number of snapshots and clones.\n\n**cStor Volumes managed via kubectl**. cStor volumes comprise of a Target Pod that exposes iSCSI. This target pod is instrumental in making the Applications using cStor Volumes portable. The cStor Target Pod then replicates the data to cStor Storage Pools. For redundancy and high-availability, the data is replicated to cStor Pools that are located on different nodes. `(Note that multiple cStor Target Pods can write to the same set of cStor Pools).`\n![cStor Volume](/pubic/images/blog/cstor-volume.png) cStor Volume\nThe cStor Target (aka cStor Volume), cStor Volume Replica and the cStor Pools are all Kubernetes custom resources. You can use the `kubectl describe` commands to check the health or events on each of these entities. And moreover, the Disks where the cStor Pools eventually write the data to, are also represented by Kubernetes Custom Resources.\n\nPlease give it a try and looking forward to your feedback. More details usage and examples are provided in the [release notes](https://github.com/openebs/openebs/releases/tag/0.8)\n\nNow, about that world map that I shared earlier, OpenEBS release v0.8 introduced the option of sending anonymous usage analytics. The analytics collected will help answer questions like:\n\n- What is a typical lifespan of a Storage Volume in the Cloud Native Environments?\n- What is the max (or average) capacity of Volumes in Kubernetes?\n- A slightly more involved question is around the nature of the Cloud Native Applications and what it means to the storage features like replication. In other words, do applications really require a storage that does replication?. Or do they just need Local PVs?.\n\nWhat was most surprising (or not) is that users don’t always wait on releases. They also tend to pick up the master builds, if it contains what they need! The above diagram depicts the Kubernetes clusters that used the OpenEBS master build before its release.\n\nOf course, you can also turn off sending the analytics if you so desire.\n","notHasFeatureImage":false,"slug":"openebs-0.8-release-allows-you-to-snapshot-and-clone-cstor-volumes"},{"id":67,"title":"Experience with OpenEBS in this Hacktoberfest","author":"Aswath","author_info":"FullStack Dev","date":"26-11-2018","tags":["OpenEBS","Open Source","Hacktoberfest"],"excerpt":"From Hacktoberfest website, I came to know that Open Source Cafe Bangalore is conducting Hacktoberfest workshops every weekend and they are also providing some awesome goodies for our contributions. ","content":"\nFrom Hacktoberfest website, I came to know that Open Source Cafe Bangalore is conducting Hacktoberfest workshops every weekend and they are also providing some awesome goodies for our contributions. So, I decided to give it a try.\n\nI was late for the workshop on day one itself (Thanks to traffic in Bangalore!). A session about `Kubernetes` was ongoing. Since I don’t even know what a `Container` is, it was hard for me to understand what `Kubernetes` is all about. The next session was about how we can make our first PR to some open source repository. Since I already made a few PRs, I understood it.\n\n## It’s better to start with simple issues first.\n\nIn most of the repositories, people will tag the issues concerning the complexity. Thanks to the [openebs](https://medium.com/@openebs) community for tagging beginner friendly issues. So, I choose to work on a few issues tagged as `hacktoberfest` — a few of them were very simple, such as fixing the linting errors. But a few of them were a little complicated like [adding autocompletion to a CLI tool](https://github.com/openebs/openebs/issues/1987).\nTaking up the more straightforward issues initially gave me confidence. And it helped me to understand the conventions to follow for that particular repository such as signing every commit that we make.\n\n## Help is available in the community if you need.\n\nWhen I decided to work on the project OpenEBS/Maya, I struggled to set up the project on my machine. I was using MacOS, and the build instructions were written for Linux OSes. People like [Satyam Zode](https://medium.com/@satyamz) and Prateek helped me to build the project by changing some lines in the `Makefile`.\nThis project was written entirely in `Golang`. When I decided to work on this project, I didn’t even know how to write a hello world program in `Golang`. But people were there to help me whenever I got stuck.\n\n## Always consult with some maintainers before you add some external libraries to the project.\n\nI took the issue of adding bash completion to a CLI tool named mayactl, since Akshay has mentioned a library to do the same thing. I learned a bit of `Golang` from A Tour of Go. Then I tried to incorporate the bash completion library to mayactl. After a couple of failed attempts, I was able to implement the essential bash completion. In the review, [Ashutosh Kumar](https://medium.com/@sonasingh46) told me that if we add an external library to our project, we should also be able to maintain it. So, he asked me to decide on it.\n\n## You don’t need to know every bit to fix an issue.\n\nInstead of using some external library, people told me to look into the source code of kubectl to see how they implemented the bash completion. I dug into kubectl’s source and figured out the part where they performed bash completion. I copied it and made it compatible with mayactl.\n\n`Copying code is fine as long as you know what it does is.`\n\nYou don’t even need to know a language to work on an issue. You can learn the language on the go. To fix the bash completion issue, all I need to know was three things:\n\n- A little bit of `Golang`\n- Know how bash completion works\n- Where to add the code inside mayactl\n\nI didn’t know all these things before I started. I learned this by doing it.\nI was happy by knowing that this PR was selected as the best PR of the week and I won an **Amazon Echo** from OpenEBS.\n\n[Tweet](https://twitter.com/proaksh/status/1056212237361115136/photo/1?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1056212237361115136&ref_url=https%3A%2F%2Fblog.openebs.io%2Fmedia%2Fa4ac13e292477c21ec22a988cdcc3daf%3FpostId%3D64b9711a22f5)\n\nThe next issue that I took was to add some features to a Common Integration Testing Framework (CITF) that OpenEBS is planning to use for all of its projects. I thought this is difficult to work on since I don’t know how OpenEBS works. But the people like Ashutosh, [Prince Rachit](https://medium.com/@princerachit) and [Akash Srivastava](https://medium.com/@srivastavaakash) explained the essential parts of OpenEBS that are required to work on the issue. Without completely understanding it, I worked on it. And with the help from the community, I could complete the tasks.\n\n## Address the review comments properly\n\nAfter completing the task, I made a PR. People approved the changes, and they asked me to put some screenshots about the feature that I added. But I was not clear about the kind of screenshot that they needed. So, I asked them, and they clarified. I made a `GIF` about adding autocompletion in mayactl and added it along with the PR.\n\n#### `In a community, help others too`\n\nWhen we were done with issues in CITF, the next task was to test the mayactl with the CITF. I added a few test cases using CITF. Then I helped others in reviewing their code and getting it merged with the master branch. I got to see that a lot of people were facing the issues that I encountered before. So, I shared the solutions that worked for me. I got happy that I also was able to help someone who was in need.\n\nFor my contributions to OpenEBS throughout this month, I won a **laptop** from them.\n\n[Tweet](https://twitter.com/openebs/status/1057711263260717056/photo/1)\n","notHasFeatureImage":false,"slug":"experience-with-openebs-in-this-hacktoberfest"},{"id":68,"title":"Running OpenEBS On Custom Rancher Cluster","author":"Chandan Sagar Pradhan","author_info":"Software Engineer at MayaData","date":"22-10-2018","tags":["CAS","Cloud Native Storage","Kubernetes","OpenEBS","Rancher"],"excerpt":"In this blog, I will be explaining how to deploy an application in a custom Rancher cluster on an OpenEBS volume.","content":"\nIn this blog, I will be explaining how to deploy an application in a custom Rancher cluster on an OpenEBS volume. **OpenEBS** is a leading open-source storage platform that provides persistent and containerized block storage for DevOps and container environments. **Rancher** is enterprise management for Kubernetes. Before you begin, please make sure all the prerequisites are met.\n\n## Node Requirements:\n\n- Ubuntu 16.04(64-bit)\n- 4 vCPUS and 16 GB RAM\n- Docker 17.03.2\n\n## Prerequisites:\n\n- Rancher 2.0 UI installed and running.\n\nAfter the installation of the docker, I used the below command to install Rancher. For more information on installing Rancher visit Rancher docs at [https://rancher.com/docs/rancher/v2.x/en/](https://rancher.com/docs/rancher/v2.x/en/)\n\n sudo docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable\n\n- Minimum of 1 master and 3 worker\n- (Optional) 6 nodes for the cluster ( 3 master and 3 workers)\n- *iscsiadm* should be present only on kubelet (And verification steps)\n\n### Remove iscsiadm from nodes:\n\nCheck the below commands on all worker nodes.\n\n iscsiadm -V\n docker exec kubelet iscsiadm -V\n\nSample Output:\n\n root@worker1 ~ # iscsiadm -V\n iscsiadm version 2.0–873\n root@worker1 ~ # sudo docker exec kubelet iscsiadm -V\n iscsiadm version 2.0–874\n\nIf your output is similar to the sample above, then you have to remove iscsi from the node. OpenEBS target will use the iscsi inside the kubelet. Run the commands below to remove iscsi from the node.\n\n service iscsid stop\n sudo apt remove open-iscsi\n\n### Load iscsi_tcp module:\n\nThe above step may remove the iscsi_tcp probe, and after a reboot, the node will not start the iscsi_tcp service, and OpenEBS volume mount will fail. It should be the same with the command below.\n\n lsmod | grep iscsi\n\nSampleOutput:\n\n root@worker113:~# lsmod | grep iscsi\n iscsi_tcp 20480 0\n libiscsi_tcp 24576 1 iscsi_tcp\n libiscsi 53248 2 libiscsi_tcp,iscsi_tcp\n scsi_transport_iscsi 98304 2 libiscsi,iscsi_tcp\n\nIf your output is similar to the sample above, then, you are good to go. If your output doesn’t have `iscsi_tcp`, you need to follow the below steps to load the `iscsi_tcp` module.\n\n modprobe iscsi_tcp\n\nYou can verify the same from the command below. Now the output should be similar to the sample output mentioned above\n\n lsmod | grep iscsi\n\n### Persist iscsi_tcp module to load after reboot:\n\nYou can make the kernel load iscsi_tcp automatically every time the node reboots by appending the line `iscsi_tcp in /etc/` modules.\n\nExample:\n\n # /etc/modules: kernel modules to load at boot time.\n #\n # This file contains the names of kernel modules that should be loaded\n # at boot time, one per line. Lines beginning with “#” are ignored.\n iscsi_tcp\n\nNow if all prerequisites have been met, go ahead with setting up the cluster. You can go directly to deploy the OpenEBS section if you already have a k8s cluster.\n\n### Creating a Custom k8s cluster on Rancher:\n\n- Once you have deployed Rancher, you should be able to access the UI. Login to your rancher UI using your credentials\n\n![rancher login screen](/images/blog/rancher-login-screen.png)\n\n- Now click on the global tab and then click on add cluster button. We should see the `add cluster` window.\n\nExample:\n![rancher ui add cluster](/images/blog/rancher-ui-add-cluster.png)\n\n- Under the add cluster option, click on the custom. Then give the cluster a name. You can customize the cluster under Cluster Options. You can choose Kubernetes version, Network provider and other options. Here, I have selected Kubernetes Version as V.1.11.2-rancher1–1, Network provider as Canal and Cloud Provider as none.\n\nExample:\n![rancher ui add custom](/images/blog/rancher-ui-add-custom.png)\n\n![rancher ui custom options](/images/blog/rancher-ui-custom-options.png)\n\n- Now, click on the Next button; it will open another page. On this page, you can select node roles like etcd/ Control Plane / Worker. Click on `Show advanced options.` Now add the i/p address of the node in the Internal Address section.\n\nExample:\n\n![rancher ui cluster summary](/images/blog/rancher-ui-cluster-summary.png)\n\n- Copy the command mentioned in the page to the node, which will be added to the cluster. Once you run the command, the page will show `1 new node registered.`\n\n![rancher ui more options](/images/blog/rancher-ui-more-options.png)\n\n- Now click on *Done*. Follow the same process to add more nodes to the cluster.\n- Once all the nodes are added, you can deploy OpenEBS on the cluster.\n\n### Deploy OpenEBS:\n\n**Note:** Ensure that you have met the prerequisites before installation.\n\nThe latest version of OpenEBS, i.e., 0.7 can be installed using the below steps.\n\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.7.0.yaml\n\n### Select Your Storage Engine:\n\nYou can now choose the storage engine to provision Jiva or cStor volumes. As a cluster admin, you can provision jiva or cStor based on your requirements.\n\nHere I am going to use the Jiva storage engine.\n\n### Provisioning Jiva Storage Engine:\n\nJiva can be provisioned in your Kubernetes cluster by using the following procedure.\n\n### Verify if the OpenEBS installation is complete.\n\nOpenEBS pods are created under `openebs` namespace, default Storage Pool and default Storage Classes are created after installation.\n\nYou can get the OpenEBS pods status by running following command\n\n kubectl get pods -n openebs\n\nYou can use the default Jiva storage class in your application YAML to run the application. You can get the storage classes that are already created by using the following command.\n\n kubectl get sc\n\n### Following is an example output.\n\n NAME PROVISIONER AGE\n openebs-cstor-sparse openebs.io/provisioner-iscsi 4m\n openebs-jiva-default openebs.io/provisioner-iscsi 4m\n openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 4m\n\nOpenEBS installation will create Jiva storage pool also. It will be created by default on `/var/openebs` inside the hosted path on the nodes.\n\nYou can get the storage pool details by running the following command.\n\n kubectl get sp\n\nFollowing is an example output.\n\n NAME AGE\n cstor-sparse-pool-gjo0 5m\n cstor-sparse-pool-str9 5m\n cstor-sparse-pool-x4dm 5m\n default 5m\n\nFrom the above output, cstor-sparse-pools are default cstor engine pools whereas *default* is the jiva storage engine default pool.\n\n### Deploying applications on OpenEBS:\n\nNow I will deploy Percona DB on OpenEBS volume.\n\nYou can get the percona deployment YAML from the command below:\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-deployment.yaml\n\nYou have to edit the percona-openebs-deployment.yaml to use the jiva storage engine.\n\nUse vi command to edit the YAML file. Inside the YAML file under the PersistentVolumeClaim section, you have to update the storageClassName. You have to use the `openebs-jiva-default` storage class.\n\nExample:\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: demo-vol1-claim\n spec:\n storageClassName: openebs-jiva-default \n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nRun the below command to deploy percona application.\n\n kubectl apply -f percona-openebs-deployment.yaml\n\nRun the below command to check percona pods should be running now.\n\n kubectl get pods\n\nExample Scenario 1:\n\n NAME READY STATUS RESTARTS AGE\n default-demo-vol1-claim-3213556361-ctrl-c96bdd757–4fhqq 2/2 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-glkbs 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-jxwvr 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-stcbb 1/1 Running 0 2m\n percona-86d6cf8547–7t6bz 1/1 Running 0 2m\n\nNow we are running an application successfully on an OpenEBS volume on a Rancher custom cluster.\n\n## Troubleshooting:\n\n**If application pod is stuck in containercreating for more time you have to follow the below steps**:\n\nExample Scenario 2:\n\n NAME READY STATUS RESTARTS AGE\n default-demo-vol1-claim-3213556361-ctrl-c96bdd757–4fhqq 2/2 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-glkbs 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-jxwvr 1/1 Running 0 2m\n default-demo-vol1-claim-3213556361-rep-58b96b64d6-stcbb 1/1 Running 0 2m\n percona-86d6cf8547–7t6bz 1/1 ContainerCreating 0 2m\n\n**Cause:**\n\n- As part of the startup of the node, iscsi was already installed and running on the node.\n- Kubelet startup doesn’t start the iscsid as it is already running on the node.\n- When volume login is initiated, iscsiadm from kubelet is trying to contact the iscsid (running on the host) to initiate the connection. The version doesn’t match that results in an error from iscsid to iscsiadm and prints `12 — module not loaded`.\n\n**Recovery Step:**\n\n- Refer to the remove iscsiadm section in pre-requisites\n\n**After rebooting the nodes, the pods will stick again in container creating state.**\n\n**Cause:**\n\n- After reboot, the kernel may not load the `iscsi_tcp` module automatically.\n\n**Recovery Step:**\n\nRefer to the persist iscsi_tcp module to load after reboot section in the pre-requisites section.\n\nHopefully, this will help you to configure OpenEBS on top of Rancher 2.0. Thank you for reading, and please provide any feedback below or on twitter — [@chandan4147](https://twitter.com/chandan4147). For more details on OpenEBS installation and troubleshooting visit [https://openebs.io/docs](/docs).\n","notHasFeatureImage":false,"slug":"running-openebs-on-custom-rancher-cluster"},{"id":69,"title":"How to install IBM Cloud Private? [Updated]","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"12-10-2018","tags":["IBM","Icp","Kubernetes","OpenEBS","Persistent Storage"],"excerpt":"In this blog, I will provide step-by-step instructions on how to configure a Kubernetes-based managed private cloud using ICP.","content":"\nIt’s been some time since I wrote about [IBM Cloud Private 2.1](http://containerized.me/introduction-to-ibm-cloud-private/), although I’ve been heavily using it in my lab.\n\nImprovements from version 1.2 to 2.1 were massive. We have also noticed the changes in the OpenEBS user community, according to our surveys usage of ICP increased dramatically. The Community Edition of the ICP 3.1 came out three weeks ago. CE container images were released with a slight delay of two weeks after the enterprise version was announced. And I believe that it deserves an updated blog to talk about the steadily advancing developer experience (and my favorite new features).\n\nIn this blog, I will provide step-by-step instructions on how to configure a Kubernetes-based managed private cloud using ICP. I will also highlight the top improvements since the last version that makes me excited.\n\n![IBM Cloud Private Architecture](https://cdn-images-1.medium.com/max/800/0*9oSUd2enJ2qhcmdk.jpg)\n\n***Quick note:*** Developers who use only public cloud ask me this question frequently, “Why would you need a private cloud, and maintain it, everything on the public cloud is much easier?”. Well, you are probably not the one paying the cloud bill, right? For various tests, (some running IO intensive workloads) and learning experiments, I maintain 4 clusters (IBM Cloud Private, Rancher, Red Hat OpenShift and one installed with kubeadm) — each 3–5 nodes. That’s 16–20 nodes, exactly $1,069.05/month — I learned it all the hard way. True that education is expensive, but running the same on a few `[*cheap servers*](https://www.ebay.com/sch/i.html?_odkw=hp+proliant+g6&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313.TR1.TRC0.A0.H0.X+HP+Proliant+DL360+G6.TRS0&_nkw=+HP+Proliant+DL360+G6&_sacat=0)` adds only ~$135/month to my electric bill — soon will be almost free thanks to the `[*solar panels*](https://www.amazon.com/gp/product/B00FF1KG8U/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B00FF1KG8U&linkCode=as2&tag=containerized-20&linkId=6304f51d54206e4f79da35d403ef4e96)` 😉`\n\n`When it’s not enough I also use StackPointCloud to deploy temporary clusters on AWS or GKE, etc, but I never keep them overnight unless I have to.`\n\n> [`On-premise turnkey solutions`](https://kubernetes.io/docs/setup/pick-right-solution/#on-premises-turnkey-cloud-solutions) `that allow me to create Kubernetes clusters on my internal, secure, cloud network with only a few commands like IBM Cloud Private makes my live really easy.`\n\nNow, let’s take a look at the requirements.\n\n#### **Prerequisites**\n\nMinimum requirements for a multi-node community edition cluster:\n\n**Hardware**\n\n- **Boot node:** 1x 1+ core(s) >= 2.4 GHz CPU, 4GB RAM, >=100 GB disk space\n- **Master node:** 1x 8+ cores >= 2.4 GHz CPU, 16+GB RAM, >=200 GB disk space\n- **Proxy node:** 1 or more 2+ cores >= 2.4 GHz CPU, 4GB RAM, >=150 GB disk space\n- **Worker node:** 1 or more 1+ cores >= 2.4 GHz CPU, 4GB RAM, >=150 GB disk space\n\nSince I’m using the Community Edition this time, I will be using single master and multiple workers configuration.\n\n**Software**\n\n- [Ubuntu 16.04 LTS](https://www.ubuntu.com/download/server) (18.04 LTS and RHEL 7.x is also supported)\n- Docker 18.03.1-ce\n- [IBM Cloud Private 3.1](https://hub.docker.com/r/ibmcom/icp-inception/)\n\n#### *How to install IBM Cloud Private-CE 3.1*\n\nIf you are already using the older version of the ICP, you can skip the cluster preparation steps and jump to the “Install IBM Cloud Private-CE 3.1” section after uninstalling the older version as described under “Uninstalling an older version of the IBM Cloud Private” section.\n\nWe need a few things installed before we get up and running with ICP 3.1. First, I’ll configure my Ubuntu servers and share SSH keys so that the master node can access all my other nodes. Then I’ll install Docker and after that ICP. From there, ICP will take care of my Kubernetes cluster installation.\n\n#### `Install the base O/S — Ubuntu (30–45mins)`\n\nDownload your preferred version of [Ubuntu](https://www.ubuntu.com/download). I use Ubuntu Server 16.04.3 LTS.\n\nInstall Ubuntu on all servers with default options. I used ***user/nopassword*** as username/password for simplicity.\n\nLog in to your Ubuntu host via terminal.\nEdit the `/etc/network/interfaces` file, assign a static IP and set a hostname.\n\nEdit the `etc/hosts` file, add your nodes to the list and make sure you can ping them by the hostname:\n`cat /etc/hosts`\n For my setup, this is how `hosts` file looks like:\n\n $ cat /etc/hosts\n 127.0.0.1 localhost\n # 127.0.1.1 icp3101# The following lines are desirable for IPv6 capable hosts\n #::1 localhost ip6-localhost ip6-loopback\n ff02::1 ip6-allnodes\n ff02::2 ip6-allrouters\n 10.10.0.161 icp3101\n 10.10.0.162 icp3102\n 10.10.0.163 icp3103\n 10.10.0.164 icp3104\n 10.10.0.165 icp3105\n\nPing nodes by hostname to make sure all are accessible:\n\n $fping icp21032 icp21033 icp21034 icp21035\n icp21032 is alive\n icp21033 is alive\n icp21034 is alive\n icp21035 is alive\n\nOn your Ubuntu host, install the SSH server:\n\n sudo apt-get install openssh-server\n\nNow, you should be able to access your servers using SSH. Check the status by running:\n\n $ sudo service ssh status\n ● ssh.service — OpenBSD Secure Shell server\n Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)\n Active: active (running) since Mon 2018–10–08 07:16:37 PDT; 8h ago\n Main PID: 1160 (sshd)\n Tasks: 1\n Memory: 6.8M\n CPU: 45.454s\n CGroup: /system.slice/ssh.service\n └─1160 /usr/sbin/sshd -D\n\nInstall *open-iscsi* and *curl* if it’s not already installed:\n\n sudo apt install curl open-iscsi\n\nRepeat above steps on all servers.\n\nNow, you need to share SSH keys among all nodes:\nLog in to your first node, which will be the boot node (ubuntu36), as root.\n\nGenerate an SSH key:\n\n ssh-keygen -b 4096 -t rsa -f ~/.ssh/master.id_rsa -N “”\n\nAdd the SSH key to the list of authorized keys:\n\n cat ~/.ssh/master.id_rsa.pub | sudo tee -a ~/.ssh/authorized_keys\n\nFrom the boot node, add the SSH public key to other nodes in the cluster:\n\n ssh-copy-id -i ~/.ssh/master.id_rsa.pub root;\n\nRepeat for all nodes.\nLog in to the other nodes and restart the SSH service:\n\n sudo systemctl restart sshd\n\nNow the boot node can connect through SSH to all other nodes without the password.\n\n#### `Install Docker CE (5mins)`\n\nTo get the latest supported version of Docker, install it from the official Docker repository.\nOn your Ubuntu nodes, update the apt package index:\n\n sudo apt-get update\n\nMake sure below packages are installed:\n\n sudo apt-get install \\\n apt-transport-https \\\n ca-certificates \\\n software-properties-common\n\nAdd Docker’s GPG key:\n\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n\nAdd the repository:\n\n sudo add-apt-repository \\\n “deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable”\n\nUpdate the apt package index:\n\n sudo apt-get update\n\nCheck the available versions:\n\n apt-cache madison docker-ce\n\nInstall Docker CE 18.0.3.1:\n\n sudo apt-get install docker-ce=18.03.1~ce-0~ubuntu\n\nMake sure it’s up and running after installation is complete:\n\n $ sudo systemctl status docker\n ● docker.service — Docker Application Container Engine\n Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)\n Active: active (running) since Mon 2018–10–08 07:17:42 PDT; 8h ago\n Docs: https://docs.docker.com\n Main PID: 1165 (dockerd)\n Tasks: 1442\n Memory: 7.2G\n CPU: 55min 10.946s\n CGroup: /system.slice/docker.service\n\n#### `Install IBM Cloud Private-CE 3.1 (60–75 mins)`\n\nDownload the IBM Cloud Private-CE container images:\n\n sudo docker pull ibmcom/icp-inception:3.1.0\n\nCreate an installation folder for configuration files and extract the sample config file:\n\n mkdir /opt/ibm-cloud-private-ce-3.1.0; \\\n cd /opt/ibm-cloud-private-ce-3.1.0\n sudo docker run -e LICENSE=accept \\\n -v “$(pwd)”:/data ibmcom/icp-inception:3.1.0 cp -r cluster/data\n\nAbove command creates the **cluster** directory under `/opt/ibm-cloud-private-ce-3.1.0` with the following files: `config.yaml`, `hosts`, and `ssh_key`. Before deploying ICP, these files need to be modified.\nReplace the `ssh_key` file with the private SSH key you have created earlier.\n\n sudo cp ~/.ssh/master.id_rsa /opt/cluster/ssh_key\n\nAdd the IP address of all your nodes to the *hosts* file in the `/opt/ibm-cloud-private-ce-3.1.0/cluster` directory. If you plan to run serious workloads, I recommend separating master and worker Kubernetes nodes. Since Community Edition supports single master node only, my config file looks like this:\n\n $ cat /opt/ibm-cloud-private-ce-3.1.0/cluster/hosts\n [master]\n 10.10.0.161\n [worker]\n 10.10.0.161\n 10.10.0.162\n 10.10.0.163\n 10.10.0.164\n 10.10.0.165\n [proxy]\n 10.10.0.161\n #[management]\n #4.4.4.4\n #[va]\n #5.5.5.5\n\nFinally, deploy the environment. Change directory to the cluster folder with the `config.yaml` file and deploy your ICP environment:\n\n sudo docker run — net=host -t -e LICENSE=accept \\\n -v “$(pwd)”:/installer/cluster ibmcom/icp-inception:3.1.0 install\n\n**Note:** I have tried to deploy 3.1 on my old VMs (where ICP v2.1.0.2 used to run) and my first attempt failed due to increased resource requirements compared to the previous version. If your deployment times-out while waiting for cloudant and you an error similar to below:\n\n TASK [addon : Waiting for cloudant to start] ******************************************\n *********************************************************************\n FAILED — RETRYING: TASK: addon :\n Waiting for cloudant to start (13 retries left).\n FAILED — RETRYING: TASK: addon :\n Waiting for cloudant to start\n\nDouble check your h/w resources and run the installer with verbose options to see more details:\n\n docker run -e LICENSE=accept — net=host \\\n -t -v “$(pwd)”:/installer/cluster \\\n ibmcom/icp-inception:3.1.0 install -vvv | tee -a install_log.txt\n\nI ended up increasing memory from 8GB to 16GB and disk capacity to 200GB (from 150GB) and all worked well after that.\n\nAfter a successful install you should see a message similar to the below:\n\n PLAY RECAP *********************************************************************\n 10.10.0.161 : ok=173 changed=94 unreachable=0 failed=0\n 10.10.0.162 : ok=113 changed=55 unreachable=0 failed=0\n 10.10.0.163 : ok=108 changed=51 unreachable=0 failed=0\n 10.10.0.164 : ok=108 changed=50 unreachable=0 failed=0\n 10.10.0.165 : ok=108 changed=49 unreachable=0 failed=0\n localhost : ok=265 changed=161 unreachable=0 failed=0POST DEPLOY MESSAGE ************************************************************The Dashboard URL: https://10.10.0.161:8443, default username/password is admin/admin\n Playbook run took 0 days, 0 hours, 53 minutes, 54 seconds\n\nOne thing I have noticed is that the installation completed much faster than the previous version. ICP got smarter and only pulling images that are required for the roles, also seems like taking advantage of the local image registries. My 5 node installation time dropped from 95 minutes to 53 minutes. If your deployment is successful, you should be able to access your ICP login screen by visiting `https://MASTERNODEIP:8443` (Default username/password is admin/admin).\n\n![account login](/images/blog/account-login.png)(***IBM Cloud Private Login Screen***)\n\n![IBM Cloud Private Dashboard](/images/blog/ibm-cloud-private-dashboard.png)(***IBM Cloud Private Dashboard***)\n\n![IBM Cloud Private Catalog](/images/blog/ibm-cloud-private-catalog.png)(***IBM Cloud Private Catalog***)\n\n#### `Uninstalling an older version of the IBM Cloud Private`\n\nIBM documentation is very clear with upgrade steps described [here](https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.0/installing/upgrade_ce.html). It is unfortunate that upgrade is only supported from the 2.1.0.3 release since I had problems installing that version. I kept my cluster at 2.1.0.2, and later clean install was the online way to go for me.\n\nI have uninstalled the existing version with the comment below:\n\n sudo docker run -e LICENSE=accept — net=host -t -v “$(pwd)”:/installer/cluster ibmcom/icp-inception:2.1.0.2 uninstall\n\nAlso removed all stopped containers:\n\n docker system prune -af\n\n#### `What’s next?`\n\nI will go over the configuration of other optional features in my next blog post as I get more familiar with the new platform.\n\n- Introduction to IBM Cloud Private #2 — What’s new in 3.1 — quick comparison vs 2.1\n- Introduction to IBM Cloud Private #3 — Catalog Applications\n- Introduction to IBM Cloud Private #4 — How to deploy workloads on OpenEBS\n- Introduction to IBM Cloud Private #5 — Monitoring IBM Cloud Private with Prometheus\n- Introduction to IBM Cloud Private #6 — Use of metrics for monitoring utilization\n- Introduction to IBM Cloud Private #7 — Contributing to the Community Charts\n- Introduction to IBM Cloud Private #8 — Chaos Engineering with Litmus\n- Introduction to IBM Cloud Private #9 — Backup your ICP cluster with OpenEBS & Heptio Ark\n\nAbove are some of the content I believe that it would be useful. If you like to see anything not covered here feel free to comment on my blog or contact me via Twitter [@muratkarslioglu](http://www.twitter.com/muratkarslioglu).\n\nTo be continued…\n\n---\n\n`Originally published at `[`Containerized Me`](http://containerized.me/how-to-install-ibm-cloud-private-updated/)`.`\n","notHasFeatureImage":false,"slug":"how-to-install-ibm-cloud-private?-[updated]"},{"id":70,"title":"Using OpenEBS for running Kubernetes stateful applications on AWS instance store disks","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"07-10-2018","tags":["Kubernetes","OpenEBS","Solutions","Stateful Applications","Tutorials"],"excerpt":"In this post, I will cover the topic of “How to set up persistent storage” using AWS instance store disks for applications running on Kubernetes clusters. ","content":"\nIn this post, I will cover the topic of “How to set up persistent storage” using AWS instance store disks for applications running on Kubernetes clusters. Instance store disks provide high performance, but they are not guaranteed to always be present in the node. This, of course, means that when the node is rescheduled you can potentially lose the data being stored. As such, there are two ways of obtaining storage on AWS for stateful applications:\n\n1. Using AWS EBS disks\n2. Using Local disks or instance store disks\n\n## Running Stateful Apps using EBS Volumes\n![Stateful Applications using EBS volumes](https://lh5.googleusercontent.com/t7qvZRLBbTPxhPJhzO6bSg6PjSD52wv2b_aDSm8G1WYai8oOc4ogMEELkN9zSmCHgY3T4XASqlguUhmLfMl0FtmCTgWD8yURvTyVZaULU9qM47L2alvFO6GQMxmd5yIenJhqHxJXRb4sHxvYNA)\n## The Problems with this Approach\n\n- When a node goes down, a new node arises as part of Auto Scaling Groups (ASG). EBS disks that are associated with the old node must be detached from the old node and attached to the new node. This process is slow and not guaranteed to work seamlessly. Also, the new EBS volume will not contain any data.\n- If the user application is capable of performing replication, then it will replicate data to this new disk. This will take more time when using large amounts of data. This will have a negative impact on performance.\n- EBS volumes are slow and users are not able to make use of faster disks such as SSDs and NVMe.\n- Slow failover means effectively no High Availability.\n- Poor I/O, unless you wish to spend a lot of unused disk space.\n\n## Why Can’t We Use Instance Disks as is for Kubernetes?\n![Stateful Applications using Instance Stores](https://lh3.googleusercontent.com/FWlk8RIHRK8tk7jvrN8jYCgV6Ho1zP8GBqiuE49y16WkaJ3gyW-8g-Z08KP0dpknnw6PNCazKMp0pCBh6AUpbJpe92Pq9Eskzb7EODaeGAYBHx4dlGBzrIX2n6iV5YMH3EFBN5hD3MycE3WeEA)\n- When a node goes down, a new node arises with its own disks, meaning any data is lost. This is because of AWS’s auto-scaling group and other policies. Per the ASG policy, the entire component associated with an instance is deleted during the termination of an EC2 instance. The user application, which has the capability to manage replication itself, has to manage the data replication across nodes.\n- What if your applications do not have this capability?\n- If a local disk fails, then your data is lost.\n\n## OpenEBS can Help Keep the Data Replicated across Nodes\n\nOpenEBS is a viable option for high availability of data, combined with the advantages of using physical disks.\n\n## How is Replication Done with OpenEBS?\n\nOpenEBS will have a minimum of 3 replicas to run the OpenEBS cluster with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk, which will come up as part of ASG. In the meantime, your workload is in accessing the live data from one of the replicas.\n![Stateful Applications using OpenEBS and Instance Stores](https://lh4.googleusercontent.com/XJCdP9q-4LX4sGr0lro6Gyj0BCMgeh85_2MTao6wnlzBAZSvtudxXplxrwDOG1zqN9n9tpltpPdHb0ssQINs2aNmQWR0M8EJRtyj30nQLMo8Zl01C-ZzTANOi0F29oUAMDvydBloOGuWme-v2w)\n## How do we Quickly Demonstrate OpenEBS on AWS?\n\n1. Setup K8s nodes to automount the disks and configure iSCSI initiators during reboot.\n2. Similarly, configure the AWS “User Data” under the launch configuration to manage the iSCSI initiators and auto mounting of disks during launch or reboot.\n3. Install OpenEBS on Kubernetes Nodes. This should be simple, and a couple of methods are discussed at the beginning of our docs, either using a Helm Chart or directly from Kubectl. More details are mentioned in [https://openebs.io/docs](/docs).\n4. Use OpenEBS Storage Classes to create Persistent Volumes for your stateful applications.\n\nBelow I provide step by step instructions that you should be able to cut, paste, and customize. As you can see, these include how to configure your AWS account for this simple POC as well.\n\n### **Summary:**\n\n> *In summary, OpenEBS can be used to easily set up stateful applications on Kubernetes where AWS instance store disks are the underlying disks. This provides good manageability, improves resilience, and allows for relatively high performance for applications.*\n\n### A detailed explanation of OpenEBS cluster deployment on AWS and rebuilding of persistent volumes are given in the section below.\n\n## Requirements\n\n- AWS account with full access to EC2, S3, and VPC\n- Ubuntu 16.04\n- KOPS tool installed\n- AWS CLI installed for AWS account access\n- SSH key generated to access EC2 instances\n\n## On Local Ubuntu Machines\n\n- Install with Ubuntu 16.04 LTS.\n- Install the kops utility package. I have followed the official[ Kubernetes documentation](https://kubernetes.io/docs/setup/custom-cloud/kops/) to complete this. I have skipped step 2 and used my VPC ID so that it can fully communicate with its components.\n- Install AWS CLI. This can be done [here](https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html).\n- Need an AWS account with full access to EC2, S3, and VPC.\n- Generate a new ssh key. If you are not familiar with this, you can check to go[ here](https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/).\n\n## Installing OpenEBS Cluster in AWS\n\n**You should have access to both the AWS management console and the local Ubuntu CLI for the installation of OpenEBS.**\n\n### Perform the following operations from the AWS management console:\n\n1. Create a Virtual Private Cloud (VPC). To do this, go to VPC service and create a VPC.\n\n![Creaate a VPC](https://lh6.googleusercontent.com/Oy2n7BoGwjRClKfVuV-_ZJh_gGH1JMV8vEf32envoIktSPNLGn8vTrB4SSkhF0-OXwVgeM0Xyo5_8qiqSn88flr9EUFyOM0AntMT78pEJqSEK2yxCm3EYKf82pHO4VYttPr35-fuMgs2lHBvTw)\n2. Create an Internet gateway and associate your VPC with this Internet Gateway. This will attach internet connectivity to your VPC. All nodes under this VPC will have outside connectivity.\n![Create an Internet gateway](https://lh3.googleusercontent.com/Yd06EwIqsr8spFLw6zwShPu0Jh53jPSrWU5evOa2YVIULFvoMG5BoV3JkbgbLLPuP2ICUXGzcJs7XlxYjN9GA241bEzY5uuppT3M0KM07m4k380AP93X5NkEXlbmVTeEgRMiCOEx3oGFFMVLXA)\n\n### **Perform the following procedure from your local ubuntu machine:**\n\n1. Download the AWS CLI utility in your local machine.\n2. Connect with your AWS account by executing the following command:\n```\n aws configure\n```\n> **Note:** You must specify your AWS Access Key, Secret Access Key, Default region name, and Default output format in order to keep the configuration details.\n\n3. Create an S3 bucket to store your cluster configuration details as follows:\n```\n aws s3 mb s3://\n``` \n\n4. Export the s3 bucket details using the following command:\n```\n export KOPS_STATE_STORE=s3://\n``` \n\n5. Create the cluster using the following command:\n```\n kops create cluster — name=.k8s.local — vpc= — zones=\n``` \n\nThis will create a cluster in the mentioned zone in your provided region as part of the AWS configuration.\n\n6. Use the set of commands produced by the above step to customize your cluster configuration, such as Cluster name change, Instance group for Nodes, and master, etc. Below is an example output.\n\n**Example:**\n\nA cluster configuration has been created.\n\nSuggestions:\n\n- list clusters with: kops get cluster\n- edit this cluster with: kops edit cluster ranjith.k8s.local\n- edit your node instance group: kops edit ig — name=ranjith.k8s.local nodes\n- edit your master instance group: kops edit ig — name=ranjith.k8s.local master-us-west-2a\n\nFinally, configure your cluster with: kops update cluster name.k8s.local — yes\n\n7. Change your instance image type and the number of machines by executing the corresponding commands. The exact command needed for your cluster will be shown at the end of the previous step. The following is an example.\n\n**Example:**\n\nChange your node configuration by executing as follows:\n```\n kops edit ig — name=.k8s.local nodes\n```\nChange your master instance type and number of machines by executing as follows:\n```\n kops edit ig — name=.k8s.local master-\n``` \n\n**Note:** We used c3.xlarge as the instance type for both Master and Nodes. The number of worker nodes used is 3 and the master node as 1.\n\n8. Once the customization is done, you can update the changes as follows:\n```\n kops update cluster .k8s.local — yes\n```\n9. The above step will deploy a 3 Node OpenEBS cluster in AWS. You can check the instance creation status by finding the EC2 instance page and choosing the corresponding region.\n\n10. From the EC2 instance page, obtain each instance type Public IP.\n\n**Example:**\n![ink](https://lh5.googleusercontent.com/MR4HVU7V-kKHlBr5J1aI8GPzVzpBwX1MAsYVCyJjIVVVatEKDXQeIPm5MMaHAe3qw9fRMnZWdOti94nqfe8C39NXI1xbQ2l9Hz30UfVFDuLd5ENPp7cZ4-mg4z2haaPWlZFFUa2wncxpsXhrMw)\n11. Go to the **Launch Configuration** section on the EC2 page and take a *copy of the Launch configuration* for nodes. Select the configuration for the Node group and click on the Actions pane.\n\n**Example:**\n![link](https://lh6.googleusercontent.com/NpHS772BGXBmArOERpQyZ2Vj3U15sEq9DAhRgutrSIZxn_yTgbVgEJPaQHYX_YNtlEwxtwWPLArSzDqxrnqqhfK8IUjgIpM6vqs94nS16n7ocHgnmg9by0CAFmN3yd_IE3Zg0I7xQzR5lYdYtw)\n12. Perform changes in the **Configure Details** section as follows:\n\na. Change the new configuration name if required.\n\nb. Edit the **Advanced Details** section and add the following entry at the end of the **User data** section.\n\n #!/bin/bash\n set -x\n date\n apt-get install open-iscsi\n grep “@reboot root sleep 120;service open-iscsi restart” /etc/crontab || sudo sh -c ‘echo “@reboot root sleep 120;service open-iscsi restart” >> /etc/crontab’\n systemctl enable open-iscsi\n sh -c ‘echo “/dev/xvdd /mnt/openebs_xvdd auto defaults,nofail,comment=cloudconfig 0 2” >> /etc/fstab’\n reboot\n set -x\n umount /mnt/openebs/xvdd\n mount /dev/xvdd /mnt/openebs_xvdd\n \n\n**Example:**\n![link](https://lh4.googleusercontent.com/4qOeijgmFhAVn1Z5zWtTlWKNwZh6GyRz5a2PvEcUafuOtzmocD6AzHb_rZ_u_A265jn87iXK085k-Qyq5Nw8EelTkdKO_g9E7SVxhIY02CnVoEXDPHXSBnJTnERQQ7bAJOme5BrGlQHT19irug)\nClick **Skip** to review and proceed with the Create launch configuration.\n\n13. Go to the **Auto Scale Group** section on the EC2 page. Select the configuration for the Node group and click on the Actions pane to edit the Launch Configuration. Change the existing one with the new Launch Configuration and save the new setup.\n\n**Example:**\n![link](https://lh5.googleusercontent.com/590yBjH_Yp97YgfGpYh4eGM-mnj2qTQr6XcjQVgzW7lzGo5-0P7fismEoBP6_XNw-2xxD4yFYPiEiqTZQk1KEEJl5TgMo9JlWEEE6tKzngV2T0H3nTu-_cZ0iXNY-UInoG6ytvlheSr8qm1eSA)\n14. SSH to each node using its public key as follows:\n```\n ssh -i ~/.ssh/id_rsa admin@\n```\n15. SSH to all Nodes where OpenEBS will be installed, and perform the following commands to install the iSCSI packages and auto mounting of the local disk during reboot.\n```\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n sudo cat /etc/iscsi/initiatorname.iscsi\n sudo service open-iscsi status\n sudo sudo sh -c ‘echo “/dev/xvdd /mnt/openebs_xvdd auto defaults,nofail,comment=cloudconfig 0 2” >> /etc/fstab’\n grep “@reboot root sleep 120;service open-iscsi restart” /etc/crontab || sudo sh -c ‘echo “@reboot root sleep 120;service open-iscsi restart” >> /etc/crontab’\n sudo reboot\n ``` \n\n16. SSH to Master Node and perform the following commands to clone the OpenEBS YAML file and deploy it.\n```\n wget\n https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml\n \n wget\n https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml\n``` \n\n17. Edit `openebs-operator.yaml` and add the following entry. This will create a storage pool on one of the local disks attached to the hosts. Refer to OpenEBS Storage Pools for more information.\n```\n — -\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: jivaawspool\n type: hostdir\n spec:\n path: “/mnt/openebs_xvdd”\n — -\n``` \n\n18. Edit `openebs-storageclasses.yaml` by adding the following entry in your corresponding storage class:\n```\n openebs.io/storage-pool: “jivaawspool”\n```\n**Example:**\n\n — -\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-percona\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/storage-pool: “default”\n openebs.io/jiva-replica-count: “3”\n openebs.io/volume-monitor: “true”\n openebs.io/capacity: 5G\n openebs.io/storage-pool: “jivaawspool”— -\n — -\n\n19. Apply `openebs-operator.yaml` by executing the following command:\n```\n kubectl apply -f openebs-operator.yaml\n```\n20. Apply openebs-storageclasses.yaml by executing the following command:\n```\n kubectl apply -f openebs-storageclasses.yaml\n```\n21. Deploy your application YAML that will be created on the local disk.\n\n**Example:**\n```\n kubectl apply -f percona-openebs-deployment.yaml\n```\n22. To check the status of applications and Jiva Pods, use the following command:\n```\n kubectl get pods -o wide\n``` \n\nAn output similar to the following should be displayed.\n\n NAME READY STATUS RESTARTS AGE IP NODE\n \n percona-7f6bff67f6-cz47d 1/1 Running 0 1m 100.96.3.7 ip-172–20–40–26.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-ctrl-84bcf764d6–269rj 2/2 Running 0 1m 100.96.1.4 ip-172–20–62–11.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-bzjq4 1/1 Running 0 1m 100.96.1.5 ip-172–20–62–11.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-lpz2k 1/1 Running 0 1m 100.96.2.8 ip-172–20–32–255.us-west-2.compute.internal\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6-rep-54b8f49ff8-rqnr7 1/1 Running 0 1m 100.96.3.6 ip-172–20–40–26.us-west-2.compute.internal\n \n\n23. Obtain the status of PVC using the following command:\n```\n kubectl get pvc\n```\nAn output similar to the following is displayed.\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n demo-vol1-claim Bound pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6 5G RWO openebs-percona 3m\n \n\n24. View the status of PV using the following command:\n```\n kubectl get pv\n```\nThe output from the above command will be similar to the following.\n\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-ef813ecc-9c8d-11e8-bdcc-0641dc4592b6 5G RWO Delete Bound default/demo-vol1-claim openebs-percona 3m\n \n\n*Now you have deployed OpenEBS with AWS local disk in your k8s environment. You will see the advantages of both low latency local disk use and fault-tolerant architecture ensured by OpenEBS.*\n\nHopefully, this helps you to better understand the benefits of using OpenEBS on top of AWS. Thank you for reading and please provide any feedback below or via Twitter — @ranjithr005\n","notHasFeatureImage":false,"slug":"using-openebs-for-running-kubernetes-stateful-applications-on-aws-instance-store-disks"},{"id":71,"title":"Contributing to mayactl cli in Hacktoberfest 2018","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"05-10-2018","tags":["OpenEBS","Minikube","Hacktoberfest","Ubuntu","Kubernetes"],"excerpt":"Hacktoberfest is an excellent platform for first-time, open-source hackers to start their journey. mayactl is a tool that allows for contribution at any level.","content":"\nHacktoberfest is an excellent platform for first-time, open-source hackers to start their journey. mayactl is a tool that allows for contribution at any level. Users can find easy fixes or find features with medium complexity that take anywhere from one to three weeks for those who are new to OpenEBS.\n\nmayactl is a command line tool designed for configuring/debugging OpenEBS. It helps with the status of various resources related to OpenEBS on a given Kubernetes cluster. For those who want to contribute to these type of features, this blog post will help you get started with mayactl. We will cover:\n\n- How to set up the development environment of OpenEBS\n- Tips on hacking mayactl\n- How to test the changes before you send a PR\n\nPre-requisites:\n\n- Beginner knowledge of Kubernetes\n- Basic knowledge of GO Language\n- Basic understanding of how OpenEBS functions. You can get started with OpenEBS [docs](/docs?__hstc=216392137.a9b75e72cb4b227999b631a7d9fb75d2.1579850476359.1579850476359.1579850476359.1&__hssc=216392137.1.1579850476359&__hsfp=3765904294) and the OpenEBS white paper\n\n## Setting up the Development Environment for mayactl\n\nSo, let’s start with the setup for the development environment of mayactl.\n\nSome pre-requisites for this:\n\n1. Any Linux system (Ubuntu recommended). For installing Ubuntu 18.04, you can follow this great article [here](https://linuxconfig.org/how-to-install-ubuntu-18-04-bionic-beaver).\n2. Golang. A very good article regarding this is available [here](https://linuxconfig.org/install-go-on-ubuntu-18-04-bionic-beaver-linux).\n3. Git. As with Ubuntu, the git tool comes preinstalled, but if it is missing, you can install it by sudo apt install git.\n4. Docker. Docker can be easily installed on Ubuntu by following the commands given below:\n\n sudo apt-get update\n sudo apt-get install apt-transport-https ca-certificates curl software-properties-common\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository \\\n \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable\"\n sudo apt-get update\n sudo apt-get install docker.io\n\n5. Open-iscsi. For installing open-iscsi on Ubuntu, use the below commands.\n\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n\n6. Kubectl. For installing kubectl on Ubuntu, you can visit this [link](https://kubernetes.io/docs/tasks/tools/install-kubectl/).\n\n7. Minikube. Install the latest version of minikube from [here](https://github.com/kubernetes/minikube/releases).\n\n## Cloning and making maya resources ready \n\nOnce you have completed the previous steps, it’s time to [fork](https://guides.github.com/activities/forking/) and [clone](https://help.github.com/articles/cloning-a-repository/) the [openebs/maya](https://github.com/openebs/maya) repo in your system. The repo should be cloned in following path `$GOPATH/src/github.com/openebs/`.\n\n mkdir -p $GOPATH/src/github.com/src/openebs\n cd $GOPATH/src/github.com/src/openebs\n git clone https://github.com//maya.git\n\nThese commands will clone maya into your local system.\n\nOnce successful, go into the maya project directory using cd maya, and run the make command in the terminal. This will build all the images of the openebs component and other important binaries, along with mayactl.\n\nOnce the command is complete, we will be ready with the mayactl binaries and other maya for testing. You can verify this by running docker images.\n\nIf the output is similar to the code above, then you are good to go.\n\n## Minikube Setup for Running a Single-Node Cluster\n\nWhat is Minikube?\n\nMinikube is a tool that enables users to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your local system (Learn [more](https://kubernetes.io/docs/setup/minikube/)).\n\nTo spin up a kubernetes cluster with minikube, you can use the following command:\n\n sudo minikube start --vm-driver none\n\nYou can verify your cluster by using the kubectl command that was previously installed. Running kubectl get nodes should generate an output similar to the one given below.\n\n NAME STATUS ROLES AGE VERSIONminikube Ready master 26m v1.10.0\n\n## Setting up OpenEBS on Minikube\n\nOnce the Kubernetes cluster is ready, it’s time to install OpenEBS. Installing OpenEBS on Kubernetes is very easy; just apply the openebs-operator to the cluster using the following command:\n\n kubectl appy -fhttps://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml.\n\nApplying the openebs-operator creates OpenEBS resources in Kubernetes. We can obtain the openebs-related resources in the OpenEBS namespace, and we can verify it by typing the following command:\n\n ashishranjan738@Ashish-PC:~$ kubectl get pods -n openebs\n \n NAME READY STATUS RESTARTS AGE\n \n cstor-sparse-pool-ni3o-8485bf44cd-fndj6 2/2 Running 0 5m\n \n maya-apiserver-674c77bb4c-dhnw5 1/1 Running 0 6m\n \n openebs-ndm-rd7hf 1/1 Running 0 6m\n \n openebs-provisioner-6d6c9ccf75-s57j8 1/1 Running 2 6m\n \n openebs-snapshot-operator-5cdd4ddc46-c2qj5 2/2 Running 0 6m\n\n## Obtaining Access to mayactl\n\nOnce all of the pods are in the Running state, and once the maya-apiserver pod is in the Running state, we can access the mayactl binary by accessing the maya-apiserver pod’s shell. We can gain access to mayactl by using the following command:\n\n kubectl exec -it -n openebs\n\n(e.g the pod name in my case is `maya-apiserver-674c77bb4c-dhnw5`). This will open the pod’s terminal over your system terminal. Typing `mayactl version` should produce something similar to this:\n\n bash-4.3# mayactl version\n \n Version: 0.7.0-unreleased\n \n Git commit: 0fa41c79dde0f6378056093d62494cd2fbb1eea4\n \n GO Version: go1.11\n \n GO ARCH: amd64\n \n GO OS: linux\n \n m-apiserver url: http://172.17.0.5:5656\n \n m-apiserver status: running\n \n Provider: KUBERNETES\n \n You can explore the current features of mayactl using the mayactl -help.command. Now we will look at where you can find the mayactl source code for hacking and how you can test the mayactl binary.\n\n## Exploring mayactl Code\n\nThe source code of mayactl can be found in cmd/mayactl in the maya [repository](https://github.com/openebs/maya/tree/master/cmd/mayactl). mayactl uses [cobra](https://github.com/spf13/cobra) cli framework for its implementation, so knowledge of this will be very helpful when starting with [good-first issues](https://github.com/openebs/openebs/labels/good%20first%20issue).\n\n## Building the mayactl Binary\n\nAfter playing with the code, it’s time to compile the code to binary and test it against the maya-apiserver. For building the mayactl binary, you can use the make mayactl command. This will trigger the build process, and the newly-built binary can found in `bin/maya` under the maya directory that you have cloned. This will generate an output of ls -ltr `bin/maya/` -ltr :\n\n## Copying the mayactl Binary to a maya-apiserver Pod\n\nOnce we are ready with the binary, we can run a test by copying it to the maya-apiserver pod’s `/tmp` folder. For copying the binary, run the following command:\n\n kubectl cp bin/maya/mayactl :/tmp -n openebs.\n\nThis will copy your mayactl binary to the maya-apiserver pod.\n\n## Testing the mayactl Binary\n\nIn order to test the mayactl binary, again get into the pod's shell using\nkubectl exec -it -n openebs. Then access the tmp directory using cd `/tmp`. You can run your new mayactl binary by prefixing the `./` before mayactl using `./mayactl` version.\n\n## A Bash Shortcut for the Above Process\n\nHmm…, so that much work is required just to test my small code change?\n\nDon’t worry, here is a shortcut that you can use to build and copy the mayactl binary to a maya-apiserver pod:\n\n make mayactl && kubectl cp bin/maya/mayactl `kubectl get pods -n openebs | grep maya-apiserver | awk {'print $1'}`:/tmp -n openebs && kubectl exec -it `kubectl get pods -n openebs | grep maya-apiserver | awk {'print $1'}` bash -n openebs\n\nRunning the above command will build the mayactl binary and copy to the maya-apiserver pod under `/tmp` directory.\n\n## Raising a PR\n\nOnce finished with the hacking, you can raise the PR to the maya repo. Before raising, however, it’s a good idea to have a look at the [contributer’s guideline](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md). Also, don’t forget to [sign](https://github.com/probot/dco/blob/master/README.md) your commit.\n\n## Joining the Slack Channel\n\nYou can reach us for any queries/announcements on our [hacktoberfest2018](https://openebs-community.slack.com/?redir=%2Fmessages%2Fhacktoberfest2018%2F) Slack channel.\n\n## Hacking begins…\n\nSo, what are you waiting for? Start your hacking now by picking some issues from [here](https://github.com/openebs/openebs/issues)! Contribution to the *hack-fest* can get you a chance to get into the *OpenEBS contributor's list* and a chance to win some exciting goodies.\n\n## Helpful References\n\n[Weekly Contributor’s Meet videos](https://www.youtube.com/watch?v=y-7mwbdVgwk&list=PLMvwgr-vV2NVdgQsU6sfFZXPMuiQGi4Hh)\n","notHasFeatureImage":false,"slug":"contributing-to-mayactl-cli-in-hacktoberfest-2018"},{"id":72,"title":"Setting up persistent volumes in RWX mode using OpenEBS","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"02-10-2018","tags":["OpenEBS","Solutions","wordpress","Nfs","Cloud Native Storage"],"excerpt":"Many stateful applications like WordPress require persistent storage in Read-Write-Many or RWX mode. OpenEBS is popular in the open-source community for its ease of use, and its simplistic design for pluggable storage engines.","content":"\nMany stateful applications like WordPress require persistent storage in Read-Write-Many or RWX mode. OpenEBS is popular in the open-source community for its ease of use, and its simplistic design for pluggable storage engines.\n\nCurrently, block volume or iSCSI support is provided natively by OpenEBS. Here, native iSCSI support means that the iSCSI stack is part of OpenEBS project and has full support for synchronous replication, granular monitoring, day 2 storage operations like backup restore, etc. iSCSI is typically used in Read-Write-Once or RWO mode which is common for all block storage engines. There have been multiple inquiries from the OpenEBS community about RWX support from OpenEBS, in other words, support for NFS. Though native support for NFS is being considered, it is already possible today to provision RWX mode storage using OpenEBS and to use it for applications like WordPress that require RXW mode.\n\n![openebs-nfs](/images/blog/wordpress-with-openebs-over-nfs.png)\n\nIn this blog, an example of WordPress is taken to show how OpenEBS storage volumes are exposed in RWX mode through the use of NFS in between WordPress and the Jiva volumes of OpenEBS.\n\nWordPress, when deployed on Kubernetes, requires both shared storage volumes on NFS and block storage volumes on iSCSI. The shared storage is required to store the core WordPress content or the admin managed content so that all the WordPress PODs can share the same data quickly. When PODs are spawned by Kubernetes on the fly to service more traffic, the PODs need to initialize quickly and require the core data to be available close to the application and in RWX mode. As an extremely high percentage of the shared storage traffic is Read traffic, the RWX volume need not be highly performant for writes and therefore the data can be served through NFS sitting in front of an iSCSI volume.\n\nThe typical deployment of a scalable WordPress application is shown in the below diagram.\n\n![wordpress-deployment-architecture](/images/blog/wordpress-deployment-architecture.png)\n\nAs shown in the above, OpenEBS can be used to serve the storage volumes in both RWO and RWX modes. The NFS storage volume for WordPress is served through the Kubernetes external storage plugin “[nfs](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs)”. The block storage for the database needs of WordPress is provided through OpenEBS JIVA volumes. If distributed databases like Percona or MariaDB are used then a common approach is that the database is deployed as a StatefulSet for horizontal scalability and Jiva volumes are deployed as a single replica. Alternatively one can use MySQL with jiva persistent volume replicating to three copies for enhanced resiliency.\n\n## Configuration details of PVC's and Storage Classes\n\n![pvc-and-storage-classes](/images/blog/pvc-and-storage-classes.png)\n\nThe PVC construct openebs-nfs-pvc and storage class construct openebs-nfs-sc are used to create an NFS share in RWX mode to be consumed by the WordPress pod. The deployment spec of nfs-provisioner uses an OpenEBS PVC claim which dynamically provisions the JIVA volumes in RWO mode and mounts them inside the nfs-provisioner pod. The entire process can take under 10–15 seconds.\n\n## A note on the required size of the NFS volume and the provisioned size of Jiva volume\n\nAs we are providing RWX volume over and above the RWO Jiva volume, iSCSI space overhead needs to be considered. It is recommended that the OpenEBS storage class specifies 10% more space than what is required by the NFS PVC.\n\nFor example, if WordPress NFS PVC spec specifies 100G as the required storage of the NFS volume, it is recommended to request 110G as the required storage from the OpenEBS Jiva volume.\n\n### TL;DR\n\nShared storage can be provisioned in RWX mode on OpenEBS through a single PVC request.\n\n## YAML spec examples:\n\nThe same YAML spec can be found in the Kubernetes external storage NFS plugin. The YAML specs below show how to provision the OpenEBS iSCSI storage in the nfs-provisioner pod automatically.\n\n## YAML spec for NFS provisioner security policy\n\nGitHub Code Snippet:\n\n \n kind: PodSecurityPolicy\n metadata:\n name: openebs-nfs-provisioner\n spec:\n fsGroup:\n rule: RunAsAny\n allowedCapabilities:\n - DAC_READ_SEARCH\n - SYS_RESOURCE\n runAsUser:\n rule: RunAsAny\n seLinux:\n rule: RunAsAny\n supplementalGroups:\n rule: RunAsAny\n volumes:\n - configMap\n - downwardAPI\n - emptyDir\n - persistentVolumeClaim\n - secret\n - hostPath\n\nBy applying the above YAML we will create PodSecurityPolicy for the NFS server.\n\n## YAML spec for ClusterRoleBinding for NFS provisioner\n\nGitHub Code Snippet:\n\n kind: ClusterRole\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-nfs-provisioner-runner\n rules:\n - apiGroups: [\"\"]\n resources: [\"persistentvolumes\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"delete\"]\n - apiGroups: [\"\"]\n resources: [\"persistentvolumeclaims\"]\n verbs: [\"get\", \"list\", \"watch\", \"update\"]\n - apiGroups: [\"storage.k8s.io\"]\n resources: [\"storageclasses\"]\n verbs: [\"get\", \"list\", \"watch\"]\n - apiGroups: [\"\"]\n resources: [\"events\"]\n verbs: [\"create\", \"update\", \"patch\"]\n - apiGroups: [\"\"]\n resources: [\"services\", \"endpoints\"]\n verbs: [\"get\"]\n - apiGroups: [\"extensions\"]\n resources: [\"podsecuritypolicies\"]\n resourceNames: [\"nfs-provisioner\"]\n verbs: [\"use\"]\n ---\n kind: ClusterRoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-run-nfs-provisioner\n subjects:\n - kind: ServiceAccount\n name: openebs-nfs-provisioner\n # replace with namespace where provisioner is deployed\n namespace: default\n roleRef:\n kind: ClusterRole\n name: openebs-nfs-provisioner-runner\n apiGroup: rbac.authorization.k8s.io\n ---\n kind: Role\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-leader-locking-nfs-provisioner\n rules:\n - apiGroups: [\"\"]\n resources: [\"endpoints\"]\n verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\"]\n ---\n kind: RoleBinding\n apiVersion: rbac.authorization.k8s.io/v1\n metadata:\n name: openebs-leader-locking-nfs-provisioner\n subjects:\n - kind: ServiceAccount\n name: openebs-nfs-provisioner\n # replace with namespace where provisioner is deployed\n roleRef:\n kind: Role\n name: openebs-leader-locking-nfs-provisioner\n apiGroup: rbac.authorization.k8s.io\n\nThe above YAML will successfully create role bindings for the NFS provisioner. So it’s time to configure NFS provisioner YAML to use OpenEBS volumes and apply them to the Kubernetes cluster.\n\n## YAML spec for NFS provisioner deployment\n\nGitHub Code Snippet:\n\n ---\n apiVersion: v1\n kind: ServiceAccount # Creating a service account for openebs-nfs-provisioner\n metadata:\n name: openebs-nfs-provisioner\n ---\n apiVersion: v1\n kind: Service # Creating a service for openebs-nfs-provisioner\n metadata:\n name: openebs-nfs-provisioner\n labels:\n app: openebs-nfs-provisioner\n spec:\n ports:\n - name: nfs\n port: 2049\n - name: mountd\n port: 20048\n - name: rpcbind\n port: 111\n - name: rpcbind-udp\n port: 111\n protocol: UDP\n selector:\n app: openebs-nfs-provisioner\n ---\n apiVersion: apps/v1\n kind: Deployment # Creating deployment for openebs-nfs-provisoner\n metadata:\n name: openebs-nfs-provisioner\n spec:\n selector:\n matchLabels:\n app: openebs-nfs-provisioner\n replicas: 1\n strategy:\n type: Recreate\n template:\n metadata:\n labels:\n app: openebs-nfs-provisioner\n spec:\n serviceAccount: openebs-nfs-provisioner\n containers:\n - name: openebs-nfs-provisioner\n image: quay.io/kubernetes_incubator/nfs-provisioner:latest\n ports:\n - name: nfs\n containerPort: 2049\n - name: mountd\n containerPort: 20048\n - name: rpcbind\n containerPort: 111\n - name: rpcbind-udp\n containerPort: 111\n protocol: UDP\n securityContext:\n capabilities:\n add:\n - DAC_READ_SEARCH\n - SYS_RESOURCE\n args:\n - \"-provisioner=openebs.io/nfs\" # Name of the provisioner\n env:\n - name: POD_IP\n valueFrom:\n fieldRef:\n fieldPath: status.podIP\n - name: SERVICE_NAME\n value: openebs-nfs-provisioner\n - name: POD_NAMESPACE\n valueFrom:\n fieldRef:\n fieldPath: metadata.namespace\n imagePullPolicy: \"IfNotPresent\"\n volumeMounts:\n - name: export-volume\n mountPath: /export\n volumes:\n - name: export-volume\n persistentVolumeClaim:\n claimName: openebspvc\n ---\n apiVersion: v1\n kind: PersistentVolumeClaim # Creating PVC for openebs-nfs-provisoner to mount on it\n metadata:\n name: openebspvc\n spec:\n storageClassName: openebs-jiva-default\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: \"110G\"\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass # Creating storage class for applications to point to openebs-nfs-provisioner\n metadata:\n name: openebs-nfs\n provisioner: openebs.io/nfs \n parameters:\n mountOptions: \"vers=4.1\" # TODO: reconcile with StorageClass.mountOptions\n\nIn the above YAML under the args users can see we are providing -provisioner=openebs.io/nfs this means that the NFS provisioner will claim volumes for those applications whose PVC is pointing this provisioner and these applications will be deployed on top of NFS provisioner, I will explain it further. Now at the end of the YAML, you can see that we are creating a PVC under which the storage class is pointing to openebs-jiva-default this means we want to use Jiva storage engine for provisioning volume for NFS provisioner.\n\n## YAML Spec for accessing NFS from WordPress\n\nGitHub Code Snippet:\n\n - -\n apiVersion: v1\n kind: Service\n metadata:\n name: wordpress-mysql\n labels:\n app: wordpress\n ....\n # Specs for the application\n ....\n volumes:\n  - name: wordpress-persistent-storage\n persistentVolumeClaim:\n claimName: openebs-nfs\n  - -\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: openebs-nfs\n annotations:\n volume.beta.kubernetes.io/storage-class: \"openebs-nfs\" # Pointing to OpensEBS-NFS-Provisioner\n spec:\n accessModes:\n   - ReadWriteMany # Making this RWX to mount on multiple applications\n resources:\n requests:\n storage: 100G\n\n## Summary\n\nOpenEBS storage can be used in RWX mode by deploying NFS provisioner in front of OpenEBS storage volume which is in RWO mode. This model is scalable for applications like WordPress. If you know or have any applications or workloads that require RWX access mode in the PVC, please comment below. Thank you for reading and connect with us at [OpenEBS Community](https://slack.openebs.io/?__hstc=216392137.d133cc61899b42c9a03d23aff802a1df.1579851978877.1579851978877.1579851978877.1&__hssc=216392137.1.1579851978878&__hsfp=3765904294)or at our Twitter handle [@openebs](http://twitter.com/openebs).\n","notHasFeatureImage":false,"slug":"setting-up-persistent-volumes-in-rwx-mode-using-openebs"},{"id":73,"title":"Using OpenEBS as the TSDB for Prometheus","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"02-10-2018","tags":["Cloud Native Storage","Kubernetes","OpenEBS","Prometheus","Solutions"],"excerpt":"Prometheus has become one of the favorite tools for monitoring metrics of applications and infrastructure in the cloud native space, especially when using Kubernetes; ","content":"\nPrometheus has become one of the favorite tools for monitoring metrics of applications and infrastructure in the cloud native space, especially when using Kubernetes; After all, Prometheus was the second project adopted by the CNCF after Kubernetes itself and in August of 2018 was the second project to graduate from CNCF as well.\n\nGiven this lineage, setting up Prometheus on Kubernetes is quite sstraightforward. You can read much more about getting going on Prometheus as well as how it compares to alternatives such as Graphite or InfluxDB on their documentation: [https://prometheus.io/docs/introduction/overview/](https://prometheus.io/docs/introduction/overview/)\n\nOne of the challenges with Prometheus is how to setup and manage the storage for it. The default behavior of Prometheus is to simply have each node store data locally however this of course exposes the user to the loss of data stored locally when the local node goes down.\n\n## Storage for Prometheus : Local and Remote\n\nPrometheus 2.4 documentation categorizes the storage for Prometheus into two types. Local storage and Remote storage. When using local storage, data monitoring data is written in TSDB format and when using Remote storage, the data is written through storage adaptors and the format in which the data is stored is not controlled by Prometheus. Here is a quick infographic about Local and Remote storage usage for Prometheus.\n![Prometheus storage — Local vs Remote](https://cdn-images-1.medium.com/max/800/0*N9cDfVd6xzRsAFK3)\nPrometheus does a great job of simplifying the data format to be written to local TSDB. However, there are some disadvantages when choosing local raw disks as the final place to store these TSDB entries, including:\n\n1. No replicated data.\n2. The data is limited to a single node. If the Prometheus server crashes or needs to be restarted for any reason, the service has to wait till it comes up on the same node.\n3. Not scalable in terms of adding more capacity or performance dynamically.\n\nBy using OpenEBS volumes as the local storage for Prometheus on Kubernetes clusters, each of the above drawbacks is addressed.\n\nBy way of introduction, OpenEBS is the easiest to use and most widely adopted open source container attached storage for Kubernetes. OpenEBS provides per application storage controller, and the storage is scalable per application. With these features and the synchronous replication capability, OpenEBS is well suited for using it as local storage for Prometheus. You can read much more about OpenEBS on [https://openebs.io/docs](/docs?__hstc=216392137.66231d5c5a305b0a901ac5f8b69a7f4c.1580127122062.1580127122062.1580127122062.1&__hssc=216392137.1.1580127122063&__hsfp=3765904294). It should take very little time to deploy and begin to use OpenEBS once you have a Kubernetes environment. Also, it may be worth noting that at MayaData in addition to sponsoring OpenEBS we also use Prometheus extensively and also WeaveCortex for our own support.\n\nThe following (mentioned [here](https://prometheus.io/docs/prometheus/latest/storage/)) is no longer true when using OpenEBS as local storage\n\n“If your local storage becomes corrupted for whatever reason, your best bet is to shut down Prometheus and remove the entire storage directory. However, you can also try removing individual block directories to resolve the problem. This means losing a time window of around two hours’ worth of data per block directory. Again, Prometheus’s local storage is not meant for durable long-term storage.”\n\n## Use OpenEBS volume as Prometheus TSDB for durability and scalability\n\n![Replicated TSDB storage for Prometheus](/images/blog/replicated-tsdb-storage-for-prometheus.png)\n\nOpenEBS volumes are replicated synchronously, in that way, data is protected and is always made available against either a node outage or a disk outage. Replica and quorum policies are independently controlled so that, for example, OpenEBS can be configured to acknowledge a write when N replicas respond, where N is <= replica count. OpenEBS can configure Kubernetes in such a way that a Prometheus pod is always scheduled in one of the three nodes that the OpenEBS replica resides. To make the data even more highly available, the OpenEBS replicas can reside in different Availability Zones of Kubernetes which in general (if in the cloud) are mapped to physical AZs of the cloud provider.\n\n## Scalability of storage\n\nThe fact that capacity cannot be added easily as needed is one of the top limitations of using local storage for Prometheus. OpenEBS helps to address this issue; capacity can be added on-demand and on-the-fly to OpenEBS volumes. So, your Prometheus local storage never runs out of storage space with OpenEBS.\n\n![On the fly storage expansion with OpenEBS](/images/blog/scalability-of-storage.png)\n\nLocal disks are grouped into storage pools, and OpenEBS volumes are carved out from these storage pools. Because of the pooling concept, you can start with small size for the storage volume and expand dynamically on demand without any service disruption i.e., on-the-fly. Disk failures are protected against an optional RAID configuration among the disks. Even an entire pool failure (can happen when a disk fails on a stripe group for example or when storage infrastructure fails) is non-fatal in OpenEBS as the rebuilding of the data happens from other replicas. So you have both local redundancy and cross-host redundancy by using OpenEBS.\n\n## What about performance? WAL support\n\nBecause OpenEBS is a pluggable, containerized architecture it can easily use different storage engines that write data to disk or underlying cloud volumes; the two primary storage engines are Jiva and cStor. Write cache support is available in the cStor storage engine. With WAL support, the write performance of Prometheus increases significantly.\n\n## How to configure OpenEBS as storage for Prometheus?\n\nJust build the storage class and update your Prometheus YAML. Your Prometheus is up and running with highly available TSDB storage with OpenEBS volumes automatically created and configured.\n\nPay attention below to the storage class parameters in the storage class. **`openebs-prometheus-sc`**\n\n ---\n # prometheus-deployment\n apiVersion: extensions/v1beta1\n kind: Deployment\n metadata:\n name: openebs-prometheus\n namespace: openebs\n spec:\n replicas: 1\n .\n .\n .\n volumeMounts:\n # prometheus config file stored in the given mountpath\n - name: prometheus-server-volume\n mountPath: /etc/prometheus/conf\n # metrics collected by prometheus will be stored at the given mountpath.\n - name: cstor-prometheus-storage-volume\n mountPath: /prometheus\n volumes:\n # Prometheus Config file will be stored in this volume\n - name: prometheus-server-volume\n configMap:\n name: openebs-prometheus-config\n # All the time series stored in this volume in form of .db file.\n - name: cstor-prometheus-storage-volume\n persistentVolumeClaim:\n claimName: cstor-prometheus-storage-volume-claim\n ---\n #PersistentVolumeClaim for prometheus\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: cstor-prometheus-storage-volume-claim\n namespace: openebs\n spec:\n storageClassName: openebs-prometheus-sc\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 300G\n\n #Use the following YAMLs to create a cStor Storage Pool.\n # and associated storage class.\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-prometheus-sc\n annotations:\n openebs.io/cas-type: cstor\n cas.openebs.io/config: |\n - name: StoragePoolClaim\n value: \"cstor-disk\"\n\n #(Optional) Below 3 lines required only if to schedule the target pods deployed on the labeled nodes\n - name: TargetNodeSelector\n value: |-\n node: appnode\n provisioner: openebs.io/provisioner-iscsi\n \n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: prometheus-pool\n spec:\n name: prometheus-pool\n type: disk\n maxPools: 3\n poolSpec:\n poolType: striped\n # NOTE - Appropriate disks need to be fetched using `kubectl get disks`\n disks:\n diskList:\n # For Eg: the below disk is from Node1 AZ1\n - disk-184d99015253054c48c4aa3f17d137b1\n - disk-2f6bced7ba9b2be230ca5138fd0b07f1\n # For Eg: the below disk is from Node2 AZ2\n - disk-806d3e77dd2e38f188fdaf9c46020bdc\n - disk-8b6fb58d0c4e0ff3ed74a5183556424d\n # For Eg: the below disk is from Node3 AZ3\n - disk-bad1863742ce905e67978d082a721d61\n - disk-d172a48ad8b0fb536b9984609b7ee653\n ---\n\n## Summary:\n\nUsing OpenEBS as storage for Prometheus on Kubernetes clusters is an easy and viable solution for production-grade deployments. Try using OpenEBS and see if it lives up to the expectation of being the easiest-to-use cloud-native storage project in Kubernetes ecosystem. Join our [slack](http://slack.openebs.io/?__hstc=216392137.66231d5c5a305b0a901ac5f8b69a7f4c.1580127122062.1580127122062.1580127122062.1&__hssc=216392137.1.1580127122063&__hsfp=3765904294) channel if you need help or to share your success story.\n","notHasFeatureImage":false,"slug":"using-openebs-as-the-tsdb-for-prometheus"},{"id":74,"title":"Storage Scheduling goes mainstream in Kubernetes 1.12","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"01-10-2018","tags":["Kubernetes","Open Source","OpenEBS","Storage Containers"],"excerpt":"With every new release of Kubernetes, I find myself in awe and also at ease with the choices we made early on to marry OpenEBS with Kubernetes.","content":"\nWith every new release of Kubernetes, I find myself in awe and also at ease with the choices we made early on to marry OpenEBS with Kubernetes.\n\nThere are a lot of Storage management capabilities being built into Kubernetes such as PV/PVC metrics, PV resize, PV Quota, Pod Priority Classes, and the Mount Propagation features that greatly enhance OpenEBS. However, I am especially excited about a couple of features that came in with Kubernetes 1.12:\n\n- Taint Nodes based on Conditions\n- Topology Aware Storage Provisioning\n\n## Taint Nodes based on Conditions ([#382](https://github.com/kubernetes/features/issues/382)):\n\nOpenEBS Volume services comprise of a Target Pod and a set of Replicas. When a node that is running the Target pod is unable to serve the Pods — the Target Pod needs to be evicted and rescheduled immediately. If you are using OpenEBS 0.6 or higher the Target Pods have the following eviction tolerations specified.\n\n - effect: NoExecute\n key: node.kubernetes.io/not-ready\n operator: Exists\n tolerationSeconds: 0\n - effect: NoExecute\n key: node.kubernetes.io/unreachable\n operator: Exists\n tolerationSeconds: 0\n\nUp until now, the above tolerations take effect only when the Kubernetes TaintNodeByCondition feature was enabled via alpha gate. With K8s 1.12, this feature has moved to beta and is enabled by default. Along with this feature, the performance improvements done in scheduling will help in faster rescheduling of the OpenEBS Target pod and so help to keep the data storage by OpenEBS highly available.\n\n## Topology Aware Dynamic Provisioning ( [#561](https://github.com/kubernetes/features/issues/561))\n\nThis feature mainly benefits the Persistent Volumes that have connectivity or access limitations such as Local PVs that cannot be accessed by Pods outside of the node or Cloud PVs like EBS and GPD that cannot be accessed outside of the zone in which they were provisioned. OpenEBS never had this limitation so this connectivity or access benefit is not really needed by the OpenEBS community.\n\nHowever, I am excited about some of the new capabilities that are now added to the StorageClass and PVC that can benefit OpenEBS volumes as well.\n\nFor instance, OpenEBS storage classes also can be set with *volumeBindingMode* of *WaitForFirstConsumer* as follows:\n\n kind: StorageClass\n apiVersion: storage.k8s.io/v1\n metadata:\n name: openebs-standard\n provisioner: openebs.io/iscsi\n volumeBindingMode: WaitForFirstConsumer\n\nThe PVCs provisioned with the above StorageClass will contain the information of the Node selected by the scheduler to launch the associated Pod in the following PVC annotation.\n\n## `volume.kubernetes.io/selected-node`\n\nOpenEBS can then use the above annotation to determine the preferred node where the Target Pod can be scheduled. This provides a simpler way to schedule the Target Pods on the same Node as the Application Pod.\n\nThis feature decidedly is an important step towards making Storage PVs a first-class citizen in scheduling. This feature helps with the initial provisioning of the volumes — I am excited about the enhancements that are planned in this area, such as the ability for the Volume Plugins to specify the preferred location where the Application Pods can be scheduled.\n\nWhile this release made progress in making Storage a first-class citizen of Kubernetes schedulers, a lot of work is underway to make the Storage Lifecycle easy to manage with the upcoming support in CSI of Snapshot, Clone, Backup and Recovery.\n\nIt feels great to be associated with Kubernetes and OpenEBS and the incredible team that is helps the DevOps teams sleep better.\n\n—\n\nBtw, it is [Hacktoberfest](https://hacktoberfest.digitalocean.com/) This is a great time to become part of the Open Source community. OpenEBS is also participating in this year’s [Hacktoberfest](/blog/celebrate-hacktoberfest-2018-with-openebs-?__hstc=216392137.073930d2db558f65dd6e9df2ff66b40e.1580119414166.1580119414166.1580119414166.1&__hssc=216392137.1.1580119414166&__hsfp=3765904294) with a friendly team that is available to help you get started with your contributions to OpenEBS and other projects.\n\nAs always, feel free to reach out to us on Slack or add comments below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.073930d2db558f65dd6e9df2ff66b40e.1580119414166.1580119414166.1580119414166.1&__hssc=216392137.1.1580119414166&__hsfp=3765904294).\n","notHasFeatureImage":false,"slug":"storage-scheduling-goes-mainstream-in-kubernetes-1.12"},{"id":75,"title":"Celebrate Hacktoberfest 2018 with OpenEBS !!","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"27-09-2018","tags":["Digital Ocean","Hacktoberfest 2018","Hacktoberfest","OpenEBS","Open Source"],"excerpt":"The 5th edition of Hacktoberfest is around the corner and it will be the second Hacktoberfest experience for the OpenEBS community. All of us are excited!","content":"\nThe 5th edition of [Hacktoberfest](https://hacktoberfest.digitalocean.com/) is around the corner and it will be the second [Hacktoberfest](https://openebs.io/hackfests/hacktoberfest-2018) experience for the OpenEBS community. All of us are excited!\n\nIn August 2017, OpenEBS community began growing and it was a beginning to building a strong foundation for an open source project. We were first introduced to Hacktoberfest by friends & peers at the DigitalOcean Bangalore Meetup and were interested in participating in it. We enlisted OpenEBS as one of the projects participating in Hacktoberfest 2017. We were pleasantly surprised by the participation and enthusiasm that Hacktoberfest attracts from developers around the world. The result was that the OpenEBS community grew by 20+ new contributor friends with ~100 PRs.\n\nWe are excited to announce that OpenEBS would be participating in Hacktoberfest 2018 as well! Look for latest updates [here](https://openebs.io/hacktoberfest)\n\n## How do you get started with Hacktoberfest?\n\nGitHub makes it really simple for contributors to get started. Just search for the tag “[hacktoberfest](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)” at the GitHub level. Within a few minutes, you can filter tens of issues to start working on that which matches your interest. First timers can look out for “good first time” tag. Anyways, here are the quick steps for you if you are interested in participating.\n\n1. Register at [DigitalOcean](https://hacktoberfest.digitalocean.com/) website and provide your mailing address so that their system knows you are participating and can send you the t-shirt and stickers later. Well, we all want them, a matter of pride 🙂\n2. Next, filter the issues with [hacktoberfest](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93) tag, and further filter out based on language, documentation, good first issue, etc\n3. Pick up few issues and let the author of the issues know that you want to fix them and start the interaction.\n4. Send a PR and with the continued interaction, you will eventually be able to get your PR merged. After sometime, your t-shirt and stickers will arrive. In the meantime enjoy working on more issues you enjoy solving.\n\n## Get additional swag from OpenEBS\n![OpenEBS swag for Hactoberfest 2018](https://cdn-images-1.medium.com/max/600/1*BXesj2ROGdmUH3vXHkRaRA.png)\nWhen your PR to any OpenEBS repositories gets merged, we will get in touch with you to fill up a form to send a special edition t-shirt designed for Hacktoberfest.\n\nWe also send some of our popular mule stickers along with your t-shirt. The best PR also gets to win a new laptop. Additional details can be found [here](https://openebs.io/hackfests/hacktoberfest-2018?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294).\n\n## Onsite workshops in Bangalore\n![Open Source Cafe at HSR Layout Bangalore](/images/blog/openebs-cafe.png)\n\nWe hope to have even greater success in this year’s edition of Hacktoberfest. With that intention, we are hosting onsite workshops on all Saturdays in October in our Open Source Cafe — Bangalore.\n\n**Schedule on the day of workshops**\n\n***10:00–10:30 AM — GitHub 101***\n\n***10:30–11:00 AM — Hacktoberfest 101***\n\n***11:00–3:00 PM — Hacktoberfest Onsite Hackathon***\n\n***11:00–12:00 PM — On-demand 30 minute sessions such as Kubernetes 101 / OpenEBS 101***\n![Onsite Hacktoberfest workshops in Bangalore](/images/blog/event-details.png)\n\nTo help beginners start their open-source journey, there will be 101-level courses in the mornings. All workshop participants will receive help from our team in getting started, and they all get to win the special edition t-shirts. Bring your own hacking gear, such as laptops!\n\n*As you will be lost in code while you are at the open-source cafe, we will bring the lunch to your hacking table :) Submit a PR on those days at the venue, collect the hacktoberfest t-shirt from the OpenEBS team, and if you order a beverage, that beer is on us. Yes, bring it on! :)*\n\n### **MULE HAS FRIENDS**\n\nThis year our theme for Hacktoberfest is “**MULE HAS FRIENDS**”. OpenEBS collaborates contributions from cloud native storage and data management. We want to drive the hacktoberfest enthusiasm to additional projects such as Weave Scope, Rancher LongHorn and Heptio ARK. This October, we are looking to help and receive more contributions for these open-source projects as well.\n\nHappy hacking!!\n\n### **Quick links**\n\n[Hacktoberfest 2018 - DigitalOcean](https://hacktoberfest.digitalocean.com/)\n\n[Hacktoberfest is a month-long celebration of open source software.](https://hacktoberfest.digitalocean.com/)\n[hacktoberfest.digitalocean.com](https://hacktoberfest.digitalocean.com/)\n\n[Build software better, together](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n\n[GitHub is where people build software. More than 28 million people use GitHub to discover, fork, and contribute to over…](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n[github.com](https://github.com/search?l=&q=state%3Aopen+label%3Ahacktoberfest&ref=advsearch&type=Issues&utf8=%E2%9C%93)\n\n[OpenEBS](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n\n[OpenEBS is an open source storage platform that provides persistent and containerized block storage for DevOps and…](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n[openebs.io](https://openebs.io/hacktoberfest?__hstc=216392137.2860b2a415ee07539ccc6f5e86f86eda.1580127992304.1580127992304.1580127992304.1&__hssc=216392137.1.1580127992304&__hsfp=3765904294)\n\n- Thanks to [Satyam Zode](https://medium.com/@satyamz?source=post_page), [Akash Srivastava](https://medium.com/@srivastavaakash?source=post_page), [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page), and [OpenEBS](https://medium.com/@openebs?source=post_page).\n","notHasFeatureImage":false,"slug":"celebrate-hacktoberfest-2018-with-openebs-!!"},{"id":76,"title":"Keeping OpenEBS volumes in RW state during “Node down” scenarios","author":"Sai Chaithanya","author_info":"A developer who is always eager to learn, loves algorithms, maths, Kubernetes, and programming, passionate about Data Science. Enjoys playing kabaddi and traveling.","date":"30-08-2018","tags":["Kubernetes","Chaos Engineering","Statefulset","OpenEBS"],"excerpt":"In this blog, I will go through a read-only issue faced at our lab in Kubernetes environment while using OpenEBS, and will also go through its possible workarounds.","content":"\nIn this post, I will go through a read-only issue experienced in the Kubernetes environment while using OpenEBS in our lab and possible workarounds.\n\nOpenEBS has the below deployment method for providing persistent storage to applications in Kubernetes clusters.\n\nAs shown in the above diagram, the application container runs on Node1, and an iSCSI target is runs on Node2 [There can be a case where the application and iSCSI target are running on the same node, but we don’t notice the issue in this scenario]. The iSCSI initiator of Node1 discovers and logs in to the iSCSI target and creates the disk `/dev/sdb`. The application consumes the mount point `/mnt/vol1` created over the disk `/dev/sdb` as persistent storage.\n\nWhen the node on which the iSCSI target, i.e. Node2, goes down, K8s then takes 5 minutes to schedule the iSCSI target on another node. This causes the mount point to enter a `Read-Only (RO)` state. Below are logs related to this issue:\n\n Aug 28 18:13:29 instance-1 kernel: [28477.898809] connection12:0: detected conn error (1020)\n Aug 28 18:15:30 instance-1 kernel: [28598.723742] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28598.730019] print_req_error: I/O error, dev sdb, sector 642376\n Aug 28 18:15:30 instance-1 kernel: [28598.737360] EXT4-fs warning (device sdb): ext4_end_bio:323: I/O error 10 writing to inode 12 (offset 186290176 size 258048 starting block 80360)\n Aug 28 18:15:30 instance-1 kernel: [28598.737364] buffer_io_error: 19448 callbacks suppressed\n Aug 28 18:15:30 instance-1 kernel: [28598.737365] Buffer I/O error on device sdb, logical block 80297\n Aug 28 18:15:30 instance-1 kernel: [28598.999756] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.006255] JBD2: Detected IO errors while flushing file data on sdb-8\n Aug 28 18:15:30 instance-1 kernel: [28599.006266] Aborting journal on device sdb-8.\n Aug 28 18:15:30 instance-1 kernel: [28599.012364] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.018784] Buffer I/O error on dev sdb, logical block 0, lost sync page write\n Aug 28 18:15:30 instance-1 kernel: [28599.028645] EXT4-fs error (device sdb): ext4_journal_check_start:61: Detected aborted journal\n Aug 28 18:15:30 instance-1 kernel: [28599.037759] EXT4-fs (sdb): Remounting filesystem read-only\n Aug 28 18:15:30 instance-1 kernel: [28599.045331] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:15:30 instance-1 kernel: [28599.052005] Buffer I/O error on dev sdb, logical block 131072, lost sync page write\n\nAs shown in the above logs, when the connection is broken the iSCSI initiator reconnects and times out after 120 seconds. This makes the mount point go into RO mode. By this time, the application containers move into the “CrashLoopback” state. To bring applications back to the Running state, a lot of manual work must be done by remounting the mount point in RW mode. It is even possible that the application could treat the host directory as a mount point and will continue writing data, which can lead to data loss or corruption.\n\nIf the iSCSI initiator had waited for 300+ seconds, the mount should have stayed in RW mode, therefore reducing the amount of manual work.\n\nWe now need to find the setting from which the iSCSI initiator obtained this timeout value. Let us first look at “iscsiadm -m session -P 3” output.\n\n iSCSI Transport Class version 2.0–870\n version 2.0–873\n Target: iqn.2016–09.com.openebs.cstor:vol1 (non-flash)\n Current Portal: 10.142.0.2:3260,1\n Persistent Portal: 10.142.0.2:3260,1\n **********\n Interface:\n **********\n Iface Name: default\n Iface Transport: tcp\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 11\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 120\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n *****\n CHAP:\n *****\n username: \n password: ********\n username_in: \n password_in: ********\n ************************\n Negotiated iSCSI params:\n ************************\n HeaderDigest: None\n DataDigest: None\n MaxRecvDataSegmentLength: 262144\n MaxXmitDataSegmentLength: 262144\n FirstBurstLength: 262144\n MaxBurstLength: 1048576\n ImmediateData: Yes\n InitialR2T: No\n MaxOutstandingR2T: 1\n ************************\n Attached SCSI devices:\n ************************\n Host Number: 1 State: running\n scsi1 Channel 00 Id 0 Lun: 0\n Attached scsi disk sdb State: running\n \n\nTo set this value, iscsid.conf has a setting called “**node.session.timeo.replacement_timeout**”. Modifying this and performing a login into the iSCSI target produces the following output for the “iscsiadm -m session -P 3” command:\n\n <>\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 13\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 300\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n <>\n \n\nHere, we can see that “Recovery Timeout” under “Timeouts” section is now shown as 300.\n\nLet’s look at kernel logs when the connection is broken:\n\n Aug 28 18:38:14 instance-1 kernel: [29963.084882] connection13:0: detected conn error (1020)\n Aug 28 18:43:15 instance-1 kernel: [30263.588590] session13: session recovery timed out after 300 secs\n Aug 28 18:43:15 instance-1 kernel: [30263.596529] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:15 instance-1 kernel: [30263.602429] print_req_error: 16 callbacks suppressed\n Aug 28 18:43:15 instance-1 kernel: [30263.602430] print_req_error: I/O error, dev sdb, sector 643248\n Aug 28 18:43:15 instance-1 kernel: [30263.609938] EXT4-fs warning (device sdb): ext4_end_bio:323: I/O error 10 writing to inode 12 (offset 186736640 size 520192 starting block 80533)\n Aug 28 18:43:15 instance-1 kernel: [30263.609943] buffer_io_error: 3951 callbacks suppressed\n Aug 28 18:43:15 instance-1 kernel: [30263.609945] Buffer I/O error on device sdb, logical block 80406\n Aug 28 18:43:16 instance-1 kernel: [30265.229113] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:16 instance-1 kernel: [30265.236187] sd 1:0:0:0: rejecting I/O to offline device\n Aug 28 18:43:20 instance-1 kernel: [30269.488365] print_req_error: I/O error, dev sdb, sector 0\n Aug 28 18:43:20 instance-1 kernel: [30269.493883] print_req_error: I/O error, dev sdb, sector 0\n Aug 28 18:43:20 instance-1 kernel: [30269.499621] Buffer I/O error on dev sdb, logical block 0, lost sync page write\n Aug 28 18:43:20 instance-1 kernel: [30269.507654] EXT4-fs error (device sdb): ext4_journal_check_start:61: Detected aborted journal\n Aug 28 18:43:20 instance-1 kernel: [30269.517548] EXT4-fs (sdb): Remounting filesystem read-only\n Aug 28 18:43:20 instance-1 kernel: [30269.524898] EXT4-fs (sdb): previous I/O error to superblock detected\n Aug 28 18:43:21 instance-1 kernel: [30269.579913] sd 1:0:0:0: rejecting I/O to offline device\n \n\nPerfect!\n\nWe notice that the iSCSI initiator is actually using this timeout setting to stop the session recovery. Also, the mount point switched to read-only after 300 seconds.\n\n**What about the sessions that are already logged into iSCSI target?**\n\nThe below command can be used to change the setting for logged-in sessions:\n\n iscsiadm -m node -T -p -o update -n node.session.timeo.replacement_timeout -v 400\n\n“`iscsiadm -m session -P 3`” output is shown below:\n\n <>\n Iface Initiatorname: iqn.1993–08.org.debian:01:9b16db669dce\n Iface IPaddress: 10.142.0.2\n Iface HWaddress: \n Iface Netdev: \n SID: 14\n iSCSI Connection State: LOGGED IN\n iSCSI Session State: LOGGED_IN\n Internal iscsid Session State: NO CHANGE\n *********\n Timeouts:\n *********\n Recovery Timeout: 400\n Target Reset Timeout: 30\n LUN Reset Timeout: 30\n Abort Timeout: 15\n ************************\n Attached SCSI devices:\n ************************\n Host Number: 1 State: running\n scsi1 Channel 00 Id 0 Lun: 0\n Attached scsi disk sdc State: running\n <>\n \n\nYou may notice the change in the “`Attached scsi disk`” value. This causes the volume to become unmounted and, therefore, needs to be remounted.\n\nIf you are aware of the iSCSI target login process, you likely know that it is a two-step process. The first step is to discover the target, and the second step is to log into the target. The “iscsiadm -o update” command can also be used after discovering the target but before logging into the target. Updating the setting in this way means that you do not need to remount the volume, as the login didn’t happen before the “iscsiadm -o update” command.\n\nBelow are the kernel logs related to the case where the iSCSI target is brought down and brought back before 400 seconds:\n\n Aug 29 11:31:37 instance-1 kernel: [19293.450614] connection12:0: detected conn error (1020)\n Aug 29 11:31:38 instance-1 iscsid: Kernel reported iSCSI connection 12:0 error (1020 — ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)\n Aug 29 11:31:41 instance-1 iscsid: connect to 10.142.0.2:3260 failed (Connection refused)\n Aug 29 11:32:26 instance-1 iscsid: message repeated 12 times: [ connect to 10.142.0.2:3260 failed (Connection refused)]\n Aug 29 11:32:29 instance-1 iscsid: connect to 10.142.0.2:3260 failed (Connection refused)\n Aug 29 11:32:37 instance-1 iscsid: message repeated 2 times: [ connect to 10.142.0.2:3260 failed (Connection refused)]\n Aug 29 11:36:38 instance-1 iscsid: connection12:0 is operational after recovery (79 attempts)\n\nSuperb!!!\n\nAs you can see in the above logs, the iSCSI connection was successful even after 300 seconds. The mount point did not go into RO state, and thus the application container will remain in the `Running` state. This avoids a lot of manual work for the user.\n\n**What about cases when the iSCSI login is already done, and the volume mountpoint can’t be remounted?**\n\nOne way to do this is by modifying the content of a file in the `/sys/class/iscsi_session/` directory path, from which the iSCSI initiator reads this setting.\n\nThe file related to this setting is as follows:\n\n /sys/class/iscsi_session/session/recovery_tmo\n\n“iscsiadm -m session -P 3” provides the session ID related to iSCSI login. It will be the “SID” parameter under the “Interface” section. For the previous output, the file to change is:\n\n /sys/class/iscsi_session/session14/recovery_tmo\n\nExecute this command to run as a root to modify this setting within the file:\n\n echo 400 > /sys/class/iscsi_session/session/recovery_tmo\n\n**Conclusion**: When Kubernetes takes time in rescheduling the iSCSI target pod into a different node, modifying this setting in any of the above methods prevents the mount point from entering the RO state.\n\n**There is one “gotcha”**: If multipathing is enabled, the multipath-related setting takes precedence over this setting if the setting modification is done either through the iscsid.conf file or “iscsiadm -o output” command.\n","notHasFeatureImage":false,"slug":"keeping-openebs-volumes-in-rw-state-during-“node-down”-scenarios"},{"id":77,"title":"OpenEBS 0.7 Release pushes cStor Storage Engine to field trials!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-08-2018","tags":["Container","Kubernetes","OpenEBS","Storage Solutions"],"excerpt":"Before I get into some fascinating features of 0.7 release, I would like to take this opportunity to thank all users who have taken our OpenEBS survey and have come forward to share your Kubernetes Stateful Workload and OpenEBS adoption stories.","content":"\nBefore I get into some fascinating features of 0.7 release, I would like to take this opportunity to thank all users who have taken our OpenEBS survey and have come forward to share your Kubernetes Stateful Workload and OpenEBS adoption stories. When I hear statements like ***“OpenEBS just works!”***, it is definitely a sign that the project and the OpenEBS Developer Community are headed in the right direction. It is both heartening and humbling. Thank you!\n![contributions survey](https://cdn-images-1.medium.com/max/800/08J2dBBRWCnbu82R_)\nWe would welcome more contributions to the survey mentioned above, for example, whether you are an OpenEBS user (yet) or not: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\n—\n\nWhile 0.6 was mainly focused towards making the storage resilient via building chaos tools like [Litmus](https://github.com/openebs/litmus); 0.7 has taken a huge leap towards bringing a new Storage Engine for your OpenEBS Volumes — called cStor. — **container** **Storage** and incidentally it is a **Storage** written in **c**)!\n\nSpeaking of OpenEBS 0.7, the geek in me would like to banter about the OpenEBS CAST (Container Attached Storage Templates or read it as moulding) framework that we developed to help easily plugin new Storage Engines, or about how we use the new Kubernetes Dynamic Client to do some heavy lifting of interacting with the Kubernetes CRDs and Scheduling. Anyways, let us keep it for the upcoming KubeCons!\n\nSo, here are some things that make OpenEBS 0.7 a significant milestone:\n\n- Increased number of Integrations\n- Node Disk Manager for Kubernetes\n- OpenEBS Control Plane\n- cStor Storage Engine\n\nIn the rest of this post, I will provide additional details on these items.\n\n—\n\n## Integrated Offerings:\n\nOpenEBS remains committed to being Easy to Setup and Use, and we are continuing our efforts to make it available from your Cluster Solutions such as KubeApps, IBM Private Cloud, StackPoint Cloud, Red Hat OpenShift, Rancher, Kontena Pharos, etc.\n\n—\n\n[Node Disk Manager (NDM)](https://github.com/openebs/node-disk-manager) for Kubernetes has made its first [release](https://github.com/openebs/node-disk-manager/releases)! It helps you better manage the Disk Inventory in your Kubernetes Cluster. The discovered block devices are added as Kubernetes custom resources called Disks. In addition to managing actual disks attached to nodes, NDM also helps developers simulate disks on their local machines using sparse files. *You can interact with NDM via kubectl.*\n\n kiran_mova@kmova-dev:~$ kubectl get disks --show-labels\n NAME AGE LABELS\n disk-cdf294e8662a87.. 2h kubernetes.io/hostname=node1...\n disk-eaf3931c96c1d9.. 2h kubernetes.io/hostname=node2...\n disk-edc04b8fcf924e.. 2h kubernetes.io/hostname=gke-kmova-\n\n To learn more about NDM and its commands, please visit\n https://github.com/openebs/node-disk-manager\n\n—\n\n## OpenEBS Control Plane (maya):\n\nAs you use OpenEBS, you realize that you almost never interact with any of the actual components of OpenEBS. This is thanks in part to the beauty of Kubernetes and how OpenEBS plugs itself into the Kubernetes orchestration layer. You can hardly notice when there is an addition to the control plane that helps provision and manage OpenEBS Volumes.\n\nTypical Storage Solutions come with a separate heavy-weight control plane, which essentially must do everything that Kubernetes does, albeit for Storage constructs like Pools and Volumes. If Storage Volumes and Pools are treated as Containers, then Kubernetes itself can Orchestrate Storage, right?\n\nThis is exactly what OpenEBS does; the Storage Volumes and Pools themselves are represented by Containers (Deployments and Services), and Kubernetes does most of the Orchestration! What is really needed by OpenEBS is to produce the Kubernetes YAMLs with the appropriate resources and scheduling constructs.\n\nThe way the Storage Pods are deployed can have specific requirements such as:\n\n- Pods should be deployed as StatefulSet or DaemonSet or regular Deployment with the custom Scheduler.\n- Pods should be associated with Node Selectors or Taints/Tolerations etc.\n- Pods should have specific privileges and/or resource limits, etc.\n- Pods should be configured with Affinity and Anti-affinity rules that make the storage highly available across node or zone failures etc.\n\nOpenEBS Release 0.7 makes it insanely easy to specify such constructs and associate them with Storage Engines, basically by just adding a few YAML files.\n\nTo understand this better, let’s observe a couple of examples.\n\n**Example 1:** Scheduling PVs of Mongo instances across Availability Zones. When Stateful Applications like Mongo are deployed, each Mongo instance gets its own PV and PVC. To be truly resilient against node failures, the PVs (actual storage) belonging to a given Mongo application must be spread out across nodes or occasionally even across availability zones. We already know how to ensure Mongo instances (pods) are spread out with Pod Anti-affinity rules.\n\nWith OpenEBS, the same Pod Anti-affinity rules can be passed on and the OpenEBS Control Plane (maya) takes care of setting them on the storage pods. All you need to do is the following:\n\n**Step 1:** Create an OpenEBS StorageClass that can be used by the PVs of Mongo StatefulSet:\n\n # Create a StorageClass suited for Mongo STS\n ---\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mongo-pv-az\n annotations:\n cas.openebs.io/config: |\n - name: ReplicaCount\n value: \"1\"\n - name: ReplicaAntiAffinityTopoKey\n value: failure-domain.beta.kubernetes.io/zone\n - name: StoragePool\n value: default\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/fstype: \"xfs\"\n\n**Step 2:** Specify the application label (say *app-unique-id*) in the Mongo StatefulSet:\n\n apiVersion: apps/v1beta1\n kind: StatefulSet\n metadata:\n name: mongo\n spec:\n serviceName: \"mongo\"\n replicas: 3\n template:\n metadata:\n labels:\n role: mongo\n environment: test\n openebs.io/replica-anti-affinity: app-unique-id\n\n**Step 3:** Specify the Storage Class in the Volume Claim Template:\n\n volumeClaimTemplates:\n — metadata:\n name: mongo-persistent-storage\n spec:\n storageClassName: mongo-pv-az\n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n\nWhen the PVs are created for the Mongo StatefulSet, the Replicas are configured with PodAntiAffinity rules with the provided application label and topology key as follows:\n\n podAntiAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n — labelSelector:\n matchLabels:\n openebs.io/replica: jiva-replica\n openebs.io/replica-anti-affinity: app-unique-id\n topologyKey: failure-domain.beta.kubernetes.io/zone\n\nThe complete YAML file for Mongo StatefulSet using OpenEBS PV is available [here](https://github.com/openebs/openebs/blob/master/k8s/demo/mongodb/mongo-statefulset.yml).\n\n**Example 2:** When running in hyper-converged mode along with applications, it is imperative to deploy granular control on how much memory or CPU can be allocated for serving a storage volume. Again, this is a relatively common scenario, and we know how to control it for the Application Deployments! The same can be extended to the Storage!\n\nFor example, you can define the Resource Requests and Limits for different storage pods:\n\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: openebs-jiva-limits\n annotations:\n cas.openebs.io/config: |\n - name: TargetResourceRequests\n value: |-\n memory: 0.5Gi\n cpu: 100m\n - name: TargetResourceLimits\n value: |-\n memory: 1Gi\n - name: ReplicaResourceRequests\n value: |-\n memory: 0.5Gi\n cpu: 100m\n - name: ReplicaResourceLimits\n value: |-\n memory: 2Gi\n cpu: 200m\n\nI have only covered a couple of examples here, but I am very excited that with the framework we have now embedded within OpenEBS, we can support any kind of scheduling requests you might have. **Bring it On!**\n\n—\n\n**cStor Storage Engine (cStor)** is the latest addition to the storage engines supported by OpenEBS. Similar to the previous Jiva/Longhorn storage engine, cStor is also completely Open Source and has no dependency on the Kernel. It can run in the platform of your choice — either on-premise or cloud!\n\nWhile cStor Volumes work just like the previous OpenEBS volumes in terms of replication and high availability, cStor has many advantages:\n\n1. Managing the Storage Space — You no longer need to worry about writing extra scripts for cleanup when Volumes are deleted.\n2. cStor reduces the number of Containers required to achieve replication capability. This is a significant shift from requiring 4 pods per Volume to just 1 new pod per Volume. The Replication is taken care of by a “Pool” pod that can serve multiple volumes.\n3. The core storage engine used to store the data is based on OpenZFS technology that has been used in production for over a decade now.\n\ncStor Pools are created on the Disks discovered by NDM. (Again, the geek in me would like to point out the excellent Operator pattern used to convert a Storage Pool Claim to Storage Pools and the cool way in which we used side-cars and CRDs to manage a non-Kubernetes storage engine in a Kubernetes way!)\n\ncStor is available as an Alpha feature with 0.7 and is very easy to try out! You can read more about cStor [here](/docs/concepts/cstor?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294#cstor).\n\nThere is a lot that I didn’t cover or address in this blog post, but I hope this served as a brief introduction to some of the most exciting features in the 0.7 release. For detailed release notes and instructions for getting started, visit [release notes](https://github.com/openebs/openebs/releases/tag/v0.7).\n\nIn my opinion, OpenEBS 0.7 decidedly will be remembered as a release that brought the coming of age of several different OpenEBS projects like [Litmus](https://github.com/openebs/litmus), [Node Disk Manager for Kubernetes (NDM)](https://github.com/openebs/node-disk-manager) and [OpenEBS cStor](/docs/concepts/cstor?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294). OpenEBS 0.7 is equipped to successfully deliver a comprehensive storage solution for DevOps persona, providing capabilities for automating Storage management in their Kubernetes Clusters.\n\nWe would welcome more contributions to the survey mentioned above, whether you are an OpenEBS user (yet) or not: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\nWhat makes this project great is your support and feedback! Please reach out to us on Slack or comment below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.c9fd9f8df74ffca9b73b72bd793ad982.1580119894211.1580119894211.1580119894211.1&__hssc=216392137.1.1580119894211&__hsfp=3765904294)/\n\nThanks to [Amit Das](https://medium.com/@amit.das?source=post_page) and [Karthik Satchitanand](https://medium.com/@karthik.s_5236?source=post_page). [Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":true,"slug":"openebs-0.7-release-pushes-cstor-storage-engine-to-field-trials!"},{"id":78,"title":"Provisioning Kubernetes cluster by kubeadm","author":"Chandan Kumar","author_info":"Software Engineer at MayaData Inc","date":"22-08-2018","tags":["Containerized Storage","Kubeadm","Kubernetes","OpenEBS","Stateful Workloads"],"excerpt":"Kubeadm is a new tool that is part of the Kubernetes distribution of 1.4.0. It allows you to install and set up a Kubernetes cluster.","content":"\n**Kubeadm** is a new tool that is part of the Kubernetes distribution of 1.4.0. It allows you to install and set up a Kubernetes cluster. One of the most frequent criticisms of Kubernetes is that it’s difficult to install. Kubeadm makes this much easier, so I strongly suggest you give it a try.\n\n### Pre-requisites for creating a cluster:\n\n- One or more machines running the compatible OS (ex: Ubuntu)\n- 2-GB or more of RAM per machine\n- 2-CPU or more for Master\n\nNetwork connectivity among all machines in the cluster\n\n### Objectives:\n\n- Install a single master Kubernetes cluster.\n- Install a Pod network on the cluster so that your Pods can communicate.\n\nBeginners can set up the pre-requisites in their own machine by creating virtual machines (VMs) in a virtual box, or they can also use multiple machines for creating clusters.\n\n### Installation:\n\nInstall these requirements in each node:\n\n- Docker\n```\n $ sudo apt-get update\n $ sudo apt-get install -y docker.io\n``` \n\n- Kubeadm, Kubelet, Kubectl\n```\n $ sudo apt-get update && sudo apt-get install -y apt-transport-https curl\n $ sudo -i\n $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -\n $ cat </etc/apt/sources.list.d/kubernetes.list\n $ deb http://apt.kubernetes.io/ kubernetes-xenial main\n $ EOF\n $ exit\n $ sudo apt-get update\n $ sudo apt-get install -y kubelet kubeadm kubectl\n``` \n\n### Master Node:\n\nThe master is the machine where the control plane components run, including etcd (the cluster database) and the API server (which the kubectl CLI communicates with).\n![Master Node](https://lh5.googleusercontent.com/BbzeYd9ttUgNLHIn-zS1gndo_sCNLYOiqe-HTHQLDNqegK72Lc7Nzg88tfUCXSo_p6Wyrq-beic2mf0ZR7tFMw3dW_IZvGYC-MT-GVCSWPqoS4OsglbwCQ2ZskBuT2FsJLI9jzY-UdokGUJnpA)\nBefore running kubeadm init in master node, first, choose a pod network add-on and verify whether it requires any arguments to be passed for kubeadm initialization. Depending on which third-party provider you choose, you might need to set the --pod-network-cidr argument with kubeadm init .\n\n- [list of pod network add-on.](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network)\n- [list of arguments](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/)\n\n### Configure the cgroup Driver used by kubelet\n```\n $ sudo sed -i \"s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g\" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n```\n### Restart kubelet\n```\n $ sudo systemctl daemon-reload\n $ sudo systemctl restart kubelet\n``` \n\nExample:\n\n```\n $ sudo kubeadm init --apiserver-advertise-address= --apiserver-cert-extra-sans=10.0.2.15 --pod-network-cidr 10.1.0.0/16\n $ sudo mkdir -p $HOME/.kube\n $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n $ sudo chown $(id -u):$(id -g) $HOME/.kube/config\n $ sudo sysctl net.bridge.bridge-nf-call-iptables=1\n $ sudo KUBECONFIG=/etc/kubernetes/admin.conf\n $ kubectl apply -f https://raw.githubusercontent.com/cloudnativelabs/kube-router/master/daemonset/kubeadm-kuberouter.yaml\n```\n\nAfter you finish running kubeadm init in master node, it provides the token, master-ip, sha and hash as follows:\n```\n $ kubeadm join --token : --discovery-token-ca-cert-hash sha256:\n```\nIf you do not have the token, you can obtain it by running the following command on the master node:\n```\n $ kubeadm token list\n```\nBy default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token using the following command:\n```\n $ kubeadm token create\n```\nFor reference, you can view this document: [https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)\n\n### Worker nodes:\n\nA worker node in Kubernetes was previously known as a minion. A node may be a VM or a physical machine, depending on the cluster. Each node has the services necessary to run pods and is managed by the master components.\n\n### Joining worker nodes:\n\nTo add nodes to your cluster, do the following for each machine:\n\n- SSH to the machine\n- Become root (e.g. sudo su -)\n- Run the command that was returned by kubeadm init. For example:\n\n $ kubeadm join — token : — discovery-token-ca-cert-hash sha256:\n\nNow you are all set and can list the nodes from the master by running\n```\n $ kubectl get nodes\n```\nPlease leave your valuable comments and questions below.\n","notHasFeatureImage":true,"slug":"provisioning-kubernetes-cluster-by-kubeadm"},{"id":79,"title":"Building persistent storage with OpenEBS and Local SSDs on GKE","author":"Ranjith Raveendran","author_info":"Ranjith is working as a Software Engineer at MayaData and working in the OpenEBS project. In his free time, he listens to music, watches movies, and goes for bike riding.","date":"21-08-2018","tags":["OpenEBS","Solutions","Stateful Applications"],"excerpt":"In this blog post, we’ll discuss the advantages of using GKE Local SSDs instead of GPDs and how to set up persistent storage using OpenEBS for highly available storage volumes.","content":"\nIf you are currently using Google Cloud for your Kubernetes needs, it is more than likely that you are using Google’s Kubernetes service, GKE. The good news is that you have another option for persistent storage apart from Google Persistent Disks, or GPDs.\n\n> *With OpenEBS software, you can easily set up persistent storage using the Local SSDs of GKE instances.*\n\nIn this blog post, we’ll discuss the advantages of using GKE Local SSDs instead of GPDs and how to set up persistent storage using OpenEBS for highly available storage volumes. Though GPDs are extremely easy to provision and natively integrated with GKE, there are some inconveniences with GPDs related to stateful applications.\n\n## Running Stateful Apps using GPD Volumes\n![GPDs as persistent disks may not be readily available](https://lh5.googleusercontent.com/JOkShLc8RiH5a-bWvFPqNWn9rs24C3toodcI3i_g7vew2-4FzYdL_-vX7X9nofXCwTMU0P9zVraY6NSRCIE5jfdlrIeXXSW7KuT74dPv7QnGn6oVRdbAKN1U9GkR-UdzDyOTPMqvoxX1EoNzIQ)\n\nWhen a GKE node goes down, a new node comes up as part of Cluster Autoscaler. GPD disks that are associated with the old node must be detached from the old node and attached to the new node. This must be done through a manual procedure. Even if automated detaching were available, it would still slow the system.\n\n- Performance of the GPD volumes is limited as they are part of a larger shared storage network. GKE is not able to make use of faster disks such as SSDs.\n- Slow fail-over essentially means no High Availability.\n- Poor I/O, unless unused disk space is not an issue.\n\n## Restrictions for Using Local SSDs as -is for Kubernetes\n\nGoogle cloud documentation on the restrictions for using local SSDs for Kubernetes is mentioned [here](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd).\n\n![Use of local disks has restrictions on GKE](https://lh4.googleusercontent.com/uE8p6fJtr2AYnqYAuv_4XWEf5ZQ1lgZ8p-wthognqxV2ayxGQf2iCh7C6LEK5qM_OwvrnDx3R4D9BaiQLac8LE1fmqm3u1BI_bDFYn2c3T-zc1cFmuvL-0WVZoAbHsv4Vo0elHttSxmdQv9OUw)\n### Here is a quick reference summarizing the restrictions.\n\nBecause local SSDs are physically attached to the node’s host virtual machine instance, any data stored exists only on that node. Since the data stored on the disks is local, your application must accommodate for this data being unavailable.\n\n[Data stored on local SSDs are ephemeral](https://cloud.google.com/compute/docs/disks/local-ssd#data_persistence). A Pod that writes to a local SSD might lose access to the data stored on the disk if the Pod is rescheduled away from that node. Additionally, if the node is terminated, upgraded, or repaired, the data will be erased.\n\nAdditionally, you cannot add local SSDs to an existing node pool.\n\n### No Restrictions with OpenEBS on the Local SSDs of GKE\n\nOpenEBS natively supports synchronous replication. This replication can be done when nodes (hence disks) are placed across Availability Zones (AZs).\n\n### Synchronous Replication of Data with OpenEBS\n\nOpenEBS uses a minimum of three replicas to run OpenEBS clusters with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk, which will come up as part of Cluster Autoscaler. In the meantime, your workload stems from accessing the live data from one of the replicas.\n![](https://lh3.googleusercontent.com/EAMk9eGAPvqcTPAd8IH_VWcfzNaSyMGEVhDDYj8CKvYqAtXc1JvmQHw_CeD-Qc_4Ny3Ahav4LjLhZgiEekcuSnij1BD2tqOEJZI-esCPP9wVsKXEiZMt-DsML0lRs6Dm4K-7LGr_XQ91IAdRfw)\n### Quick Steps to Set Up OpenEBS on GKE\n\n- Set up a three-node GKE cluster with local disks by enabling the Cluster Autoscaler feature.\n- Install OpenEBS on Kubernetes Nodes. This should be simple, and a couple of methods are discussed at the beginning of our docs, using either a Helm Chart or directly from Kubectl. More details are mentioned in the [OpenEBS documentation](/docs).\n- Use OpenEBS Storage Classes to create Persistent Volumes for your stateful applications.\n\n## Detailed Explanation of OpenEBS 0.7 Cluster Deployment on GKE across AZs and Rebuilding of PVs.\n\n### Prerequisites\n\n- GKE account with credentials to create instances using the local disk.\n- Ubuntu 16.04\n- gcloud CLI access\n\n### Creating a GKE Cluster\n\nYou can start by entering your login information to your Google Cloud Platform and create the GKE cluster as follows:\n\n1. Click “CREATE CLUSTER” from the console and fill in the required details.\n\n![](https://lh3.googleusercontent.com/jjAzX7Vykgt0ChGM5MdQ8K6cEdxiB9-3xWO309T4hSizm__EnWsH_pO61lgJZQAbDEJim9K0OmfKYu7IgqhUg_BhYw94YinMDbA72GvkoC7Vmzzgrw4q0rqYfOSbNqIV-F16ONCkJa1lN0e6gw)\n2.  Click “Advanced Edit” under the Node Pool section and Enable the “Auto Scaling” feature for the cluster. Select the Node Image type as “Ubuntu.”\n![](https://lh3.googleusercontent.com/ud5ky3UMDH8GL1FuOLrXSaVC-9q1-3bdYjEPzHcPM_uhWc_h8NIGa1s2yKuUHhAe1dLo388jGupB8mVcYfBIDo9zS0ABqOzU9YxclXvaoHTtXprtAjArXUNh4IoEgirTw21wiMCIKAyQZCGZQQ)\n3.  Add the required number of local SSD disks per Node as follows and save the changes.\n![](https://lh4.googleusercontent.com/6PJNdbLybtKYi7vQrBHjCmoH8AjYNFh-FaMXX0AEpm0KyFfsL3lwTHMuF1H1s0P8UasWmC7kk2_83iJXYM0_mHYfJ2HSwo58iGAbOoP94an6GdlJwjAIYTmqdza0Bj-IknbZaSraTokLtaTGIA)\n4. We have deployed the cluster across different AZs. For high availability, and in case one zone goes down, the data and application would remain intact. You can select the required zones by clicking on “Advanced options”. I have chosen the following zones rather than my default zone.\n![](https://lh3.googleusercontent.com/n8sUfShxkqLsL5btIo1RucZftcalGmbbZNxtaNqhjhT4CxkeowKdY1yhB-BF5TbBWHfnqMo3YAWWuhCGwpBfXZLq2m00rAeEshnzvOSEp0Xz3Z9k0qXOacurtXL8HDDgnAnd5_BRWFVYSkjNEA)\n5. Now you can create your K8s cluster by clicking the “Create” button.\n\n6. Check your Kubernetes Cluster status under the “Kubernetes Clusters” section.\n\n7. Once your cluster is ready, launch the gcloud shell to access your cluster using CLI.\n\n8. You may need an admin-context if you are installing OpenEBS without Helm. Use the following command to set the admin-context:\n ```\n kubectl create clusterrolebinding -cluster-admin-binding --clusterrole=cluster-admin --user=\n ```\n9. Before installing OpenEBS, you will need to select one of the mounted SSD disks to create an OpenEBS Storage Pool. You must note the mount path of the selected disk, as we need to provide the mount path details while creating the storage pool.\n\nIn my cases, I have chosen two local SSDs per node, and it is mounted automatically on the Node as part of the instance creation. Among these disks, I will choose the “sdb” disk. The OpenEBS Jiva storage pool will be created on the “sdb” disk on each node. Steps for creating pools are mentioned below.\n\n You can obtain the mounted disk details on Nodes using the following command:\n\n sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL\n\n Below is an example output.\n\n NAME FSTYPE SIZE MOUNTPOINT LABEL\n sdb ext4 375G /mnt/disks/ssd0\n sdc ext4 375G /mnt/disks/ssd1\n sda 100G\n └─sda1 ext4 100G / cloudimg-rootfs\n\n10. You are now ready to install the OpenEBS 0.7 cluster using the following command. Then follow the steps below from your master node.\n\n ```\n kubectl apply -f https://openebs.github.io/charts/openebs-operator-0.7.0.yaml\n\n ```\n\n11. You can check the OpenEBS running pod details using the following command:\n\n kubectl get pods -n openebs\n\nBelow is an example output:\n\n NAME READY STATUS RESTARTS AGE\n cstor-sparse-pool-9stx-5b6556cc54–4q249 2/2 Running 0 22s\n cstor-sparse-pool-ew8c-54bd645b6c-6s2d5 2/2 Running 0 22s\n cstor-sparse-pool-fml8–8685fbbc74-dg4fs 2/2 Running 0 22s\n maya-apiserver-85657db5cf-b7sr9 1/1 Running 0 1m\n openebs-ndm-857lr 1/1 Running 0 57s\n openebs-ndm-gpxh6 1/1 Running 0 57s\n openebs-ndm-tjmhg 1/1 Running 0 57s\n openebs-provisioner-7594dbf8cd-ph8jn 1/1 Running 0 59s\n openebs-snapshot-operator-7dddbf56db-7xtct 2/2 Running 0 58s\n \n\n12. You can then check the default storage classes created as part of the OpenEBS operator installation using the following command:\n\n ```\n kubectl get sc\n ```\nAgain, here is an example output:\n\n NAME PROVISIONER AGE\n openebs-cstor-sparse openebs.io/provisioner-iscsi 1h\n openebs-jiva-default openebs.io/provisioner-iscsi 1h\n openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter 1h\n standard (default) kubernetes.io/gce-pd 1h\n \n\n13. Create a storage pool on an external disk mounted on the nodes. In step 8, I mentioned using the “sdb” disk for creating an OpenEBS storage Pool. To create a storage pool, create a file called “openebs-config.yaml” in your master node and add the example YAML (shown below) while changing the appropriate mounted disk path.\n\nFor example, if your external disk is mounted as `/mnt/disks/ssd0` in your nodes, change the path as shown below.\n\n path: “/mnt/disks/ssd0”\n\n Example yaml file:\n\n ---\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: default\n type: hostdir\n spec:\n path: \"/mnt/disks/ssd0\"\n ---\n \n14. Apply the modified *openebs-config.yaml* file by using the following command:\n\n ```\n kubectl apply -f openebs-config.yaml\n ```\n\nThis will create a storage pool called “default” on the selected disk.\n\n15. The storage pool is now created on the Nodes according to your requirement. You can now get the storage pool details by running the following command:\n\n ```\n kubectl get sp\n ```\n\n Example output:\n\n NAME AGE\n cstor-sparse-pool-9stx 1h\n cstor-sparse-pool-ew8c 1h\n cstor-sparse-pool-fml8 1h\n default 1h\n \n\n16. Now your configured OpenEBS Jiva storage engine will create an OpenEBS Jiva volume on the storage pool established on your local SSD disk.\n\n17. Deploy your application YAML, which will be consumed in the OpenEBS Jiva volume.\n\nWe now have the Percona deployment application YAML and need to change the storage class mentioned in the YAML with the default storage class name “openebs-jiva-default” for the Jiva volume. This will be created as part of the *openebs-operator.yaml* installation.\n\n18. We now use the following Percona deployment YAML file to deploy a stateful Percona application using the Jiva volume. Obtain the YAML file using following command:\n```\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/percona/percona-openebs-deployment.yaml\n```\n\n19. Edit the downloaded “percona-openebs-deployment.yaml” and conduct the following changes.\n\nIn the “PersistentVolumeClaim” section, under metadata, add the following content.\n\n labels:\n \n “volumeprovisioner.mapi.openebs.io/replica-topology-key-domain”: “failure-domain.beta.kubernetes.io” \n “volumeprovisioner.mapi.openebs.io/replica-topology-key-type”: “zone” \n\nNow change the “storageClassName” from ”openebs-standard” to “openebs-jiva-default”.\n\n20. Save the modified changes and apply the YAML as follows. This will create a PVC and PV in the mentioned size.\n\n\n kubectl apply -f percona-openebs-deployment.yaml\n\n\n21. You can then view the PVC status by running the following command:\n\n kubectl get pvc\n\n Below is an example output.\n\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n \n demo-vol1-claim Bound default-demo-vol1-claim-2300073071 5G RWO openebs-jiva-default 11m\n\n\n22. You can then obtain the PV status by running the following command:\n\n kubectl get pv\n\nAnd here is an example output.\n\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n default-demo-vol1-claim-2300073071 5G RWO Delete Bound default/demo-vol1-claim openebs-jiva-default\n \n\n23. Now, your Percona application pod will be running along with three Jiva volume replicas and one Jiva Controller pod. You can view the running pod status by running the following command:\n\n\n kubectl get pods -o wide\n\n\nWhich will provide the following output.\n\n NAME READY STATUS RESTARTS AGE IP NODE\n default-demo-vol1-claim-2300073071-ctrl-78d4fbc66d-v4454 2/2 Running 0 14m 10.44.2.9 gke-mayadata-gke-default-pool-2d42a511-7gcz\n default-demo-vol1-claim-2300073071-rep-79f97d5494-cvd8z 1/1 Running 0 14m 10.44.0.8 gke-mayadata-gke-default-pool-5eed8925-7391\n default-demo-vol1-claim-2300073071-rep-79f97d5494-g2hff 1/1 Running 0 14m 10.44.1.8 gke-mayadata-gke-default-pool-54c3be93-qsgl\n default-demo-vol1-claim-2300073071-rep-79f97d5494-qqvv4 1/1 Running 0 14m 10.44.2.10 gke-mayadata-gke-default-pool-2d42a511-7gcz\n percona-7f6bff67f6-mjp9d 1/1 Running 0 14m 10.44.1.9 gke-mayadata-gke-default-pool-54c3be93-qsgl\n \n\n### Summary\n\n![OpenEBS on GKE](https://lh5.googleusercontent.com/3zgM48Nep0Uszwxu_mIfewSP2SXCSdWHTeSzA7JbJfgaQnbaxB8SVd5gF6ADpXmmnnd84NqctMrm7CfFlkDuhSTPsuLCWxRm9dgtSXI6bOh-Gl_oKsbkubON_To3QcDbHXJA20n909Jie0lLXQ)\n> *Local SSDs on your GKE cluster can be used as persistent storage for stateful applications such as Prometheus, WordPress, MongoDB, etc. This gives you the advantages of both the low latency local SSD use and fault-tolerant architecture ensured by OpenEBS.Thank you for reading. Feel free to join our *[*Slack*](https://slack.openebs.io/)* channel for any questions or help needed, or to share your success stories of using OpenEBS on GKE.*\n","notHasFeatureImage":false,"slug":"building-persistent-storage-with-openebs-and-local-ssds-on-gke"},{"id":80,"title":"cStor Pool Provisioning in OpenEBS 0.7","author":"Ashutosh Kumar","author_info":"Software Engineer at MayaData | OpenEBS Reviewer and Contributor | CKA | Gopher | Kubernaut","date":"19-08-2018","tags":["Kubernetes","OpenEBS","Pool","Storage"],"excerpt":"OpenEBS team is happy to announce the release of 0.7 which comes with a new storage engine for creating storage pool known as cStor engine.","content":"\nGreetings OpenEBS users!\n\nThe OpenEBS team is happy to announce the release of 0.7, which comes with a new storage engine, known as cStor, for creating storage pools.\nTo find out more details on this specific release, please visit the following links:\n[https://github.com/openebs/openebs/releases](https://github.com/openebs/openebs/releases)\n[https://openebs.io/blog/openebs-07-release-pushes-cstor-storage-engine-to-field-trials](/blog/openebs-07-release-pushes-cstor-storage-engine-to-field-trials)\n\nTo keep this story short and concise I will jump directly into how you can provision a storage pool in 0.7 using the cStor engine. Just for your information, storage pools can also be provisioned using the Jiva engine that we used in previous versions of OpenEBS.\n\nLet’s get started!\n\nThere are currently two ways to provision a storage pool using the cStor engine in OpenEBS 0.7. As we move forward with the tutorial, I will assume that you already have a Kubernetes cluster set up. I will follow up this tutorial by having a 3-node Kubernetes cluster with one physical disk attached to every node on GKE.\n\n### Manual Pool Provisioning\n\nTo provision a storage pool manually, you run the following command where your kubernetes cluster is configured.\n\n ashutosh@miracle:~$ kubectl get disk\n NAME CREATED AT\n disk-26ac8d634b31ba497a9fa72ae57d6a24 1d\n disk-2709a1cba9cea9407b92bc1f7d1a1bde 1d\n disk-427145375f85e8a488eeb8bbfae45118 1d\n sparse-4b488677f76c94d681870379168a677a 1d\n sparse-c3ddc8f0de2eb17c50d145cf6713588c 1d\n sparse-e09fe4b5170a7b8fd6b8aabf8c828072 1d\n\nThe output with prefix disk represents your physical disks, and the output with prefix sparse represents sparse disks. We will discuss the sparse disk concept in a later blog-post, but for now, let’s concentrate on physical disks! We need to simply copy the physical disks (in the diskList field of the storage pool claim YAML) over which the pool should be created. The SPC YAML will look like the following:\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk\n spec:\n name: cstor-disk\n type: disk \n poolSpec:\n poolType: striped\n disks:\n diskList:\n - disk-26ac8d634b31ba497a9fa72ae57d6a24\n - disk-2709a1cba9cea9407b92bc1f7d1a1bde\n - disk-427145375f85e8a488eeb8bbfae45118\n\nNow you just need to apply the created YAML. That’s it. Done!\n\n ashutosh@miracle:~$ kubectl apply -f spc.yaml \n storagepoolclaim.openebs.io/cstor-disk created\n ashutosh@miracle:~$ kubectl get sp\n NAME CREATED AT\n cstor-disk-5wsi 4s\n cstor-disk-7pgs 4s\n cstor-disk-8fhi 4s\n default 14d\n\nEach disk that we entered in SPC YAML is attached to a specific node. So, 3 cStor pools were created on top of the 3 nodes, and this sp (the output belongs to the applied SPC) belongs to the applied SPC. If all 3 disks were attached to a single node, we would only have one sp.\n\n### Dynamic Pool Provisioning\n\nThe steps listed above involved a heavy manual process, but it helps the user to configure a storage pool based on their choices. If you didn’t like that manual process, don’t worry; let’s work some magic here by doing something known as dynamic pool provisioning.\n\nStart by applying the following SPC YAML :\n\n apiVersion: openebs.io/v1alpha1\n kind: StoragePoolClaim\n metadata:\n name: cstor-disk-dynamic\n spec:\n name: cstor-disk-dynamic\n type: disk\n # required in case of dynamic provisioning\n maxPools: 3 \n # If not provided, defaults to 1 (recommended but not required)\n minPools: 3\n poolSpec:\n poolType: striped\n ashutosh@miracle:~$ kubectl apply -f dynamic_spc.yaml storagepoolclaim.openebs.io/cstor-disk-dynamic created\n ashutosh@miracle:~$ kubectl get sp\n NAME CREATED AT\n cstor-disk-dynamic-jwc5 6s\n cstor-disk-dynamic-qot0 6s\n cstor-disk-dynamic-s8va 6s\n default 14d\n\nThat’s it. Done!\n\nLet’s better understand what’s going on here:\n\n1. The dynamic way of pool provisioning will support reconciliation, i.e. the OpenEBS control plane will always try to get the maxPools number of pools specified on storagepoolclaim. If the SPC YAML is applied and the node is down or there are no existing disks, when the resources come up, the pool will be provisioned automatically without any intervention.\n2. The manual method of provisioning will not have any such reconciliation.\n3. The number of the pool(s) specified by minPools will be created or no pools will be provisioned. So, if maxPool=10 and minPool=6, the Control plane will always try to get to a pool count of 10 but any single shot of provisioning in any one part of the reconciliation loop must provision at least 6 pools. Once the minPools count is reached even if the count of pool increases by only 1, the control plane will execute the action.\n\n### NOTES: \n\n1. In the above tutorial, we provisioned a striped type of pool for both cases. A mirrored pool can also be provisioned, but at least 2 disks must be attached to the node. Simply change the poolType field in SPC YAML to mirrored.\n2. The number of sp resources created is equal to the number of cStor pools created on top of each node, but those belong to a single SPC that spawns them. For applications, the pool is virtually one while we create volumes.\n\nHopefully, this helps! Feel free to post any queries or concerns and share any feedback. You can reach out on our OpenEBS slack channel as well.\n([https://slack.openebs.io/](https://slack.openebs.io/))\n","notHasFeatureImage":false,"slug":"cstor-pool-provisioning-in-openebs-0.7"},{"id":81,"title":"How to start contributing to mayactl","author":"Sumit Lalwani","author_info":"Sumit Lalwani is a Software Engineer at Mayadata. He is a Kubernetes enthusiast and passionate about open source, containers, cloud, and arm. He loves to learn and code.","date":"14-08-2018","tags":["Docker","OpenEBS","Kubernetes"],"excerpt":"mayactl is the command line tool for interacting with OpenEBS volumes. mayactl is not used/required while provisioning or managing the OpenEBS volumes, but it is currently used while debugging and troubleshooting.","content":"\n## What is mayactl?\n\n- mayactl is the command line tool for interacting with OpenEBS volumes. mayactl is not used/required while provisioning or managing the OpenEBS volumes, but it is currently used while debugging and troubleshooting.\n- mayactl is the client like kubectl which requests to maya-apiserver to get specific information whereas kubectl requests to Kubernetes apiserver to get specific information.\n- mayactl helps retrieve storage related information for debugging/troubleshooting storage related issues. mayactl provides various commands to create volume, get volume details and create, list and revert snapshot and many more.\n\nTo know more about the mayactl visit:\n\n[https://openebs.io/docs/deprecated/mayactl](/docs/deprecated/mayactl)\n\n## OpenEBS Architecture\n\n![OpenEBS Architecture](/images/blog/openebs-architecture.png)\n\nTo know about the OpenEBS visit: [https://openebs.io/docs](/docs)\n\n### These few things are required to be installed in your system (with Ubuntu host) to run mayactl\n\n1. **Docker**\n\n- To install docker run these commands\n\n ```\n sudo apt-get update\n sudo apt-get install apt-transport-https ca-certificates curl software-properties-common\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository \\\n \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n $(lsb_release -cs) \\\n stable\"\n sudo apt-get update\n sudo apt-get install docker.io\n ```\n\n- Or you can visit the official docker website to install docker\n\n[https://openebs.io/docs/deprecated/mayactl](/docs/deprecated/mayactl)\n\n2. **open-iscsi package**\n\n- To install open-iscsi package run these commands\n\n ```\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n ```\n\n3. **Golang**\n\n- To install golang, visit the official golang website: [https://golang.org/doc/install](https://golang.org/doc/install)\n\n4. **minikube**.\n\n- To install minikube, run these commands:\n\n ```\n # minikube requires virtualbox to be installed as a dependency\n\n sudo apt-get install virtualbox virtualbox-ext-pack\n sudo apt-get update\n\n # minikube version 0.24.0\n\n curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.24.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/\n ```\n\n5. **Kubectl**.\n\n- Run these commands to install Kubectl:\n\n ```\n curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.0/bin/linux/amd64/kubectl\n chmod +x ./kubectl\n sudo mv ./kubectl /usr/local/bin/kubectl\n ```\n\n### How do I run mayactl in a local machine for development purposes?\n\n1. Open the openebs repo ( [https://github.com/openebs/openebs](https://github.com/openebs/openebs)) and star the openebs repo.. 😄 (Not Mandatory)\n\n2. Fork the openebs/openebs and openebs/maya repositories into your GitHub account.\n\n- Visit and click on the fork option (both repositories)\n\n[a) openebs/openebs](https://github.com/openebs/openebs)\n\n[OpenEBS is containerized block storage written in Go for cloud native and other environments w/ per container (or pod)…](https://github.com/openebs/openebs)[github.com](https://github.com/openebs/openebs)\n\n[b) openebs/maya](https://github.com/openebs/maya)\n\n[maya — OpenEBS Maya extends Kubernetes capabilities to orchestrate CAS containers.](https://github.com/openebs/maya)[github.com](https://github.com/openebs/maya)\n\n3. Clone the openebs and maya repositories inside your gopath. Then run these commands to clone:\n\n```\n# if directories not present create the directories in same hierarchy\n\ncd $GOPATH/src/github.com/openebs\ngit clone https://github.com//openebs.git\ngit clone https://github.com//maya.git\n```\n\n4. Run the single node cluster using the minikube command.\n\n```\nminikube start --vm-driver=none\n\n# To check whether minikube is configured and running\n\nminikube status\n```\n\n5. Install OpenEBS by executing these commands:\n\n```\ncd $GOPATH/src/github.com/openebs/openebs/k8s/\nkubectl apply -f openebs-operator.yaml\nkubectl apply -f openebs-storageclasses.yaml\n```\n\n![kubernetes commands](/images/blog/install-openebs-by-commands.png)\n\n6. Now we have the openebs-provisioner and maya-apiserver running as a pod in the Kubernetes (minikube) cluster.\n\n- To get the pods, run this command:\n\n ```\n kubectl get pods\n NAME READY STATUS RESTARTS AGE\n maya-apiserver-7b8d5496cc-kgmnn 1/1 Running 0 3m\n openebs-provisioner-6797d44769-phnnc 1/1 Running 2 3m\n ```\n\n7. To run/access mayactl, you will need to login/execute into the maya-apiserver pod on Kubernetes.\n\n- Find out the name of the maya api-server pod by running the following commands:\n\n ```\n kubectl get pods\n\n # It will access the bash shell inside the pod\n\n kubectl exec -it /bin/bash\n ```\n\n8. Now you can run all mayactl commands as you are inside the maya-apiserver pod.\n\n- Try running these commands after exececute/login into the pod.\n\n ```\n mayactl -help\n ```\n\nGo through the issues ([https://github.com/openebs/maya/issues](https://github.com/openebs/maya/issues)) and start modifying the mayactl code, located in $GOPATH/src/github.com/openebs/maya/cmd/mayactl, and start contributing to OpenEBS. Also, you can start contributing by writing a small unit test code in mayactl. For every PR you raise, you will also receive goodies from the OpenEBS team. 😃\n\n**How do I test the changes made in mayactl?**\n\n1. After modifying the mayactl code, go into the maya directory, i.e $GOPATH/src/github.com/openebs/maya, and run these commands:\n\n```\n# run this if not currently in maya directory\n\ncd $GOPATH/src/github.com/openebs/maya\n\n# this will create the mayactl binary into the bin folder inside maya directory\n\nmake mayactl\n```\n\n2. After the build has been completed, copy the mayactl binary from the bin folder to the maya-apiserver pod using the command:\n\n```\nkubectl cp $GOPATH/src/github.com/openebs/maya/bin/maya/mayactl :/tmp/\n```\n\n3. Login/execute into the maya-apiserver pod to run the mayactl binary.\n\n```\nkubectl exec -it /bin/bash\ncd /tmp/\n```\n\n4. Here the mayactl binary you copied is shown.\n\n- To run that binary, use the following command:\n\n ```\n ./mayactl -help\n ```\n\nNow you can easily see the changes you made in the mayactl command line tool. You are also now ready to raise the PR’s. 😃\n\n**Reference:**\n\n- [https://www.youtube.com/watch?v=yzMEYT-yzRU](https://www.youtube.com/watch?v=yzMEYT-yzRU)\n- [https://openebs.io/docs](/docs?__hstc=216392137.2d2e61b1b8f85b3675bfaef604437f8a.1580205967521.1580205967521.1580205967521.1&__hssc=216392137.1.1580205967522&__hsfp=2262740235)\n","notHasFeatureImage":false,"slug":"how-to-start-contributing-to-mayactl"},{"id":82,"title":"Provisioning Google Cloud with k8s using it’s in-house tool, KOPS","author":"Harshvardhan Karn","author_info":"Harshvardhan Karn works at MayaData Inc. He is a public speaker, has talked in few local meetups and at two major conferences. In his free time, he likes to play Guitar, Netflix.","date":"14-08-2018","tags":["Docker","Kubernetes","OpenEBS","Kops"],"excerpt":"Setting up and using a cluster in GCP offers a few significant advantages over using GKE. For instance, using GCP gives the user the liberty to use their custom binaries or a pure Open Source Kubernetes.","content":"\nI am very excited for this post, as I have been working on this content for a few weeks. I might sound naive throughout this, mainly because I am, to be honest. Setting up and using a cluster in _GCP_ offers a few significant advantages over using _GKE_. For instance, using GCP gives the user the liberty to use their _custom binaries_ or a _pure Open Source Kubernetes_. Also, _GKE_ does not support modification or access to the master node, whereas a manually setup-ed k8s cluster over _VMs_ does.\n\nTo gain the full understanding of Kubernetes, some people like to get their hands dirty for a course of interval. Luckily, we have a few more tools to get a Kubernetes cluster up and running in VM instances of **Google Cloud**. I find that tools like _kubernetes-incubator/kubespray , crosscloudci/cross-cloud_ and kops, are not very straightforward to use, but _kops, kubespray_ is somewhat close. _Cross-cloud_ on the other hand, has poor documentation and is not very stable. _kops_ here stands for ‘_Kubernetes Operations._’ To be honest, I find this to be a good tool to deploy the Cluster over Google Cloud Platform (**GCP**) or Amazon Web Services (**AWS**). I certainly would not say it is the best, but this tool is documented to the extent that one can use it. The original idea of KOPS was to create user a production ready cluster in **AWS**. Allowing it to provision the GCP is luxury since GCP already provides with GKE and AWS does not.\n\n### **Requirements**\n\nThese items are required to deploy the production-ready k8s cluster in GCP:\n\n- **KOPS**\n\n ```\n wget -O kops\n https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '\"' -f 4)/kops-linux-amd64\n\n chmod +x ./kops\n\n sudo mv ./kops /usr/local/bin/\n ```\n\n- **GCloud**\n\n [https://cloud.google.com/sdk/](https://cloud.google.com/sdk/)\n\n Once you are done installing GCloud SDK, you must run _gcloud init_. This will configure your gcloud with your existing GCP project.\n\n- **kubectl**\n\n From the [Official Kubernetes kubectl release:](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n\n ```\n wget -O kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl\n chmod +x ./kubectl\n sudo mv ./kubectl /usr/local/bin/kubectl\n ```\n\n- A little patience …\n\n## Let’s Begin!\n\nA quick note: every time you create a cluster, it also creates a _Virtual Private Cloud_ (**VPC**), per se. Google Cloud allows you to create only a maximum of 5 VPC’s in one project, and a total of only 5 clusters. So, to resolve this problem, we can create a **VPC** explicitly and use it as a common Network for the rest of the clusters.\n\n### Create a VPC\n\nHere I am using subnet-mode as auto, and it will create a VPC _openebs-e2e_ with a subnet in every zone.\n\n```\ngcloud compute networks create openebs-e2e --project=openebs-ci --subnet-mode=auto\n```\n\n### Create a Bucket\n\nKops needs a State Store to hold the configuration of our cluster. In our case, it is Google Cloud Storage Buckets. So, let’s create one empty Bucket using the following:\n\n```\ngsutil mb gs://openebs-dev/\n```\n\nNow, since we are ready with the Bucket, we can populate it with our cluster’s State Store, i.e. Cluster object and InstanceGroup object.\n\n**Create the Cluster & InstanceGroup Objects in Our State Store**\n\n_kops create cluster_, creates the Cluster object and InstanceGroup object. Here, we’ll be working within kops.\n\n```\nPROJECT=`gcloud config get-value project`\nexport KOPS_FEATURE_FLAGS=AlphaAllowGCE # to unlock the GCE features\nkops create cluster openebs-dev.k8s.local --zones us-central1-a\n--state gs://openebs-dev/ --project=${PROJECT}\n--kubernetes-version=1.11.1 --node-count 3\n```\n\nNow we can list the Cluster objects in our kops State Store (the GCS bucket we created):\n\n```\nkops get cluster --state gs://openebs-dev/\nNAME CLOUD ZONES\nopenebs-dev.k8s.local gce us-central1-a\n```\n\n**NB:** It is not necessary to use the same name for the Bucket and Cluster; you are free to use whatever name you wish.\n\n### Create a Cluster\n\nWe are now ready with all of the changes and the cluster configuration, so we will proceed with the creation of the cluster. _kops create cluster_ created the Cluster object and the InstanceGroup object in our State Store, but it did not actually create any instances or other cloud objects in GCE. To do that, we’ll use _kops update cluster_.\n\n_kops update cluster_ without _--yes_ will show us a preview of changes that will be made. It comes handy in case we want to see or verify the specs before creation.\n\n```\nkops update cluster openebs-dev.k8s.local --state gs://openebs-dev/ --yes\n```\n\nCheers!\n\nWe have now deployed the Kubernetes cluster on GCP. If you go to the _Compute Engine_ in _Google Cloud Platform_, you will find 4 new nodes, where 1 is the master and the rest are worker nodes. Just to save your day, if you are wondering why you could not find your nascent cluster inside _Google Kubernetes Engine_, this is not a mistake or error because it is not a GKE Cluster. All GzCP knows is that there are 4 VMs running in the project, which we know is a K8s cluster.\n\n## Out-of-the-Box\n\nWe are now ready with the cluster, but is it ready for the deployments? Once the kops is finished creating the cluster, we can validate its readiness using the following:\n\n```\nkops validate cluster --state gs://openebs-dev/\n\n\nI0808 12:34:10.238009 25907 gce_cloud.go:273] Scanning zones: [us-central1-c us-central1-a us-central1-f us-central1-b]\nINSTANCE GROUPS\nNAME ROLE MACHINETYPE MIN MAX SUBNETS\nmaster-us-central1-a Master n1-standard-1 1 1 us-central1\nnodes Node n1-standard-2 3 3 us-central1\nNODE STATUS\nNAME ROLE READY\nmaster-us-central1-a-067f master True\nnodes-6rt6 node True\nnodes-lvs5 node True\nnodes-wbb8 node True\nYour cluster openebs-dev.k8s.local is ready\n```\n\nIf you find that the cluster not ready, wait for a few minutes as it takes some time to configure the cluster. You can even check using _kubectl_ from your control machine:\n\n```\nkubectl get nodes\n```\n\nYou will see the node counts once your Cluster is up, viz. _kubelets_ are configured. If you are wondering how you got your _kubectl_ configured to this cluster, _kops_ does that for you. It exports a kubecfg file for a cluster from the state store to your _~/.kube/config_ local machine where you are running _kops_. If you want to export this config to some other path, you can the following:\n\n```\nkops export kubecfg openebs-dev.k8s.local\n```\n\nI wrote an Ansible playbook for [Litmus](https://github.com/openebs/litmus/), which is actually a wrapper for all of these to bring up the cluster on GCP. You can check it out here:\n\n[https://github.com/openebs/litmus/tree/master/k8s/gcp/k8s-installer](https://github.com/openebs/litmus/tree/master/k8s/gcp/k8s-installer)\n\nThe playbook also checks the cluster availability implicitly using a python script. This will hold the playbook from termination until the cluster is ready to use. _kops validate_ works well, but **not** for **k8s version < 1.9,** up to the day of writing this post.\n\nGodspeed!\n","notHasFeatureImage":false,"slug":"provisioning-google-cloud-with-k8s-using-it’s-in-house-tool,-kops"},{"id":83,"title":"Setting up WordPress and SQL with OpenEBS","author":"Ashish Ranjan","author_info":"An enthusiastic person when it comes to software & computers. I don't mind getting out of my comfort zone when things related to computing need to be done at the spur of the moment.","date":"14-08-2018","tags":["OpenEBS","Kubernetes","Cloud Native Storage","Open Source","State Department"],"excerpt":"Wordpress is a well-known blogging platform. New bloggers are often surprised when they find out how easy it is to get set up and start their first piece in this popular tool.","content":"\n[Wordpress](https://en.wikipedia.org/wiki/WordPress) is a well-known blogging platform. New bloggers are often surprised when they find out how easy it is to get set up and start their first piece in this popular tool. In this blog, we will show how to deploy WordPress and MySQL on OpenEBS in their Kubernetes cluster.\n\n## What is OpenEBS?\n\nOpenEBS offers containerized persistent block storage using Docker containers. Those blocks are often referred to as Virtual Storage Machines (similar to K8s pods). OpenEBS seamlessly provides scalable storage volumes and manages them effortlessly. For more information, you can visit [https://openebs.io/join-our-slack-community](https://openebs.io/join-our-slack-community?__hstc=216392137.b7acacf689e0cc4579eea008f86d0c72.1579857743065.1579857743065.1579857743065.1&__hssc=216392137.1.1579857743066&__hsfp=3765904294).\n\n### Prerequisites:\n\n- A k8s cluster with at least one minion.\n- Basic knowledge of writing services, deployment in k8s.\n- Kubectl, already configured.\n- A code editor for writing a yamls.\n- Brains.\n\nLet’s get started!\n\n## Setting up OpenEBS\n\nBefore starting with WordPress, we need to set up OpenEBS. For this article, I will be using OpenEBS v0.6 (you are free to use newer versions if you wish).\n\nWhen setting up OpenEBS, you need to apply the following yamls:\n\n```\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-operator.yaml\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/v0.6/k8s/openebs-storageclasses.yaml\n```\n\nThe first yaml is for the openebs-operator, and the second one is for openebs-storage-classes. For more information look [here](/docs?__hstc=216392137.b7acacf689e0cc4579eea008f86d0c72.1579857743065.1579857743065.1579857743065.1&__hssc=216392137.1.1579857743066&__hsfp=3765904294). After applying the above yamls, the output of kubectl get pods — all-namespaces will look like this:\n\n```\n$ kubectl get pods --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\nkube-system event-exporter-v0.2.1-5f5b89fcc8-bhv7r 2/2 Running 0 16m\nkube-system fluentd-gcp-scaler-7c5db745fc-tb2zw 1/1 Running 0 16m\nkube-system fluentd-gcp-v3.0.0-wqgzx 2/2 Running 0 14m\nkube-system heapster-v1.5.3-77c6fcd568-q8txc 3/3 Running 0 15m\nkube-system kube-dns-788979dc8f-4lgsf 4/4 Running 0 16m\nkube-system kube-dns-autoscaler-79b4b844b9-jldbr 1/1 Running 0 16m\nkube-system kube-proxy-gke-ashish-ranjan-default-pool-b3a38b91-cv5d 1/1 Running 0 16m\nkube-system l7-default-backend-5d5b9874d5-hvrgb 1/1 Running 0 16m\nkube-system metrics-server-v0.2.1-7486f5bd67-hls82 2/2 Running 0 15m\nopenebs maya-apiserver-68c98fdb76-vbslv 1/1 Running 0 1m\nopenebs openebs-provisioner-5569654c96-hmhb5 1/1 Running 0 1m\nopenebs openebs-snapshot-operator-5f7c4d9bd8-7fnfv 2/2 Running 0 1m\n```\n\nWait until all openebs namespaced pods move into a running state. Once this is completed, we’ll start by creating a secret for sql.\n\n## Creating a Secret\n\n```\nkubectl create secret generic mysql-pass --from-literal=password=w0rdPres5\n```\n\nRun the above kubectl command to create a mysql password.\n\n## Wordpress Deployment\n\nNow, let's start writing the WordPress deployment yaml. Copy and save the above into a Wordpress.yaml file and execute a kubectl apply on it. Once this is done, the output of kubectl get pods,svc,pvc — all-namespaces will look similar to this:\n\n```\n$ kubectl get pods,svc,pvc --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-766678794-jtltg 2/2 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-2pkt4 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-htdxh 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-rdtlk 1/1 Running 0 2m\ndefault pod/wordpress-7bdfd5557c-5b4nh 1/1 Running 2 2m\nkube-system pod/event-exporter-v0.2.1-5f5b89fcc8-wprbm 2/2 Running 0 8m\nkube-system pod/fluentd-gcp-scaler-7c5db745fc-s9mp9 1/1 Running 0 8m\nkube-system pod/fluentd-gcp-v3.0.0-49vq6 2/2 Running 0 5m\nkube-system pod/fluentd-gcp-v3.0.0-kfjsx 2/2 Running 0 5m\nkube-system pod/fluentd-gcp-v3.0.0-tg5hh 2/2 Running 0 5m\nkube-system pod/heapster-v1.5.3-76f7f5f544-z7xk9 3/3 Running 0 6m\nkube-system pod/kube-dns-788979dc8f-2chf4 4/4 Running 0 8m\nkube-system pod/kube-dns-788979dc8f-ls4ln 4/4 Running 0 7m\nkube-system pod/kube-dns-autoscaler-79b4b844b9-8745z 1/1 Running 0 8m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8866 1/1 Running 0 7m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8fh5 1/1 Running 0 7m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-9jjf 1/1 Running 0 8m\nkube-system pod/l7-default-backend-5d5b9874d5-k9mn8 1/1 Running 0 8m\nkube-system pod/metrics-server-v0.2.1-7486f5bd67-wlnlg 2/2 Running 0 6m\nopenebs pod/maya-apiserver-68c98fdb76-5bt2z 1/1 Running 0 4m\nopenebs pod/openebs-provisioner-5569654c96-4n4hp 1/1 Running 0 4m\nopenebs pod/openebs-snapshot-operator-5f7c4d9bd8-shk58 2/2 Running 0 4m\nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndefault service/kubernetes ClusterIP 10.55.240.1 443/TCP 8m\ndefault service/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.253.18 3260/TCP,9501/TCP 2m\ndefault service/wordpress LoadBalancer 10.55.253.170 104.154.224.12 80:31392/TCP 2m\nkube-system service/default-http-backend NodePort 10.55.243.156 80:30045/TCP 8m\nkube-system service/heapster ClusterIP 10.55.248.200 80/TCP 8m\nkube-system service/kube-dns ClusterIP 10.55.240.10 53/UDP,53/TCP 8m\nkube-system service/metrics-server ClusterIP 10.55.244.66 443/TCP 8m\nopenebs service/maya-apiserver-service ClusterIP 10.55.240.121 5656/TCP 4m\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\ndefault persistentvolumeclaim/wp-pv-claim Bound pvc-7030ae5f-9d40-11e8-afcb-42010a800179 20Gi RWO openebs-standard 2m\n```\n\n## MySql deployment\n\nFollow the same procedure as done for WordPress deployment and execute a kubectl apply on it. The output of kubectl get pods,svc ,pvc — all-namespaces will look similar to this:\n\n```\n$ kubectl get pods,svc,pvc --all-namespaces\nNAMESPACE NAME READY STATUS RESTARTS AGE\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-ctrl-b5c4f588f-lrl2l 2/2 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-2jcgx 1/1 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-j5tsk 1/1 Running 0 2m\ndefault pod/pvc-082a54c8-9d41-11e8-afcb-42010a800179-rep-66d7f6fb46-q9kww 1/1 Running 0 2m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-766678794-jtltg 2/2 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-2pkt4 1/1 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-htdxh 1/1 Running 0 6m\ndefault pod/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-rep-6689868cf4-rdtlk 1/1 Running 0 6m\ndefault pod/wordpress-7bdfd5557c-5b4nh 1/1 Running 4 6m\ndefault pod/wordpress-mysql-bcc89f687-zlt5q 1/1 Running 0 2m\nkube-system pod/event-exporter-v0.2.1-5f5b89fcc8-wprbm 2/2 Running 0 11m\nkube-system pod/fluentd-gcp-scaler-7c5db745fc-s9mp9 1/1 Running 0 11m\nkube-system pod/fluentd-gcp-v3.0.0-49vq6 2/2 Running 0 9m\nkube-system pod/fluentd-gcp-v3.0.0-kfjsx 2/2 Running 0 9m\nkube-system pod/fluentd-gcp-v3.0.0-tg5hh 2/2 Running 0 9m\nkube-system pod/heapster-v1.5.3-76f7f5f544-z7xk9 3/3 Running 0 10m\nkube-system pod/kube-dns-788979dc8f-2chf4 4/4 Running 0 11m\nkube-system pod/kube-dns-788979dc8f-ls4ln 4/4 Running 0 10m\nkube-system pod/kube-dns-autoscaler-79b4b844b9-8745z 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8866 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-8fh5 1/1 Running 0 11m\nkube-system pod/kube-proxy-gke-ashish-ranjan-default-pool-f0958e58-9jjf 1/1 Running 0 11m\nkube-system pod/l7-default-backend-5d5b9874d5-k9mn8 1/1 Running 0 11m\nkube-system pod/metrics-server-v0.2.1-7486f5bd67-wlnlg 2/2 Running 0 10m\nopenebs pod/maya-apiserver-68c98fdb76-5bt2z 1/1 Running 0 7m\nopenebs pod/openebs-provisioner-5569654c96-4n4hp 1/1 Running 0 7m\nopenebs pod/openebs-snapshot-operator-5f7c4d9bd8-shk58 2/2 Running 0 7m\nNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndefault service/kubernetes ClusterIP 10.55.240.1 443/TCP 12m\ndefault service/pvc-082a54c8-9d41-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.251.84 3260/TCP,9501/TCP 2m\ndefault service/pvc-7030ae5f-9d40-11e8-afcb-42010a800179-ctrl-svc ClusterIP 10.55.253.18 3260/TCP,9501/TCP 6m\ndefault service/wordpress LoadBalancer 10.55.253.170 104.154.224.12 80:31392/TCP 6m\ndefault service/wordpress-mysql ClusterIP None 3306/TCP 2m\nkube-system service/default-http-backend NodePort 10.55.243.156 80:30045/TCP 11m\nkube-system service/heapster ClusterIP 10.55.248.200 80/TCP 11m\nkube-system service/kube-dns ClusterIP 10.55.240.10 53/UDP,53/TCP 11m\nkube-system service/metrics-server ClusterIP 10.55.244.66 443/TCP 11m\nopenebs service/maya-apiserver-service ClusterIP 10.55.240.121 5656/TCP 7m\nNAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\ndefault persistentvolumeclaim/mysql-pv-claim Bound pvc-082a54c8-9d41-11e8-afcb-42010a800179 20Gi RWO openebs-standard 2m\ndefault persistentvolumeclaim/wp-pv-claim Bound pvc-7030ae5f-9d40-11e8-afcb-42010a800179 20Gi RWO openebs-standard 6m\n```\n\nIf all of the pods are running, check the external IP of the WordPress load balancer. For example, in my case it is **104.154.224.12**. Open the your web browser IP if you are redirected to your WordPress setup page. Congratulations! Your WordPress is now ready for you to start your blog!\n\nHappy blogging!\n\n### Proof:\n\n![wordpress](/images/blog/wordpress.png)\n","notHasFeatureImage":false,"slug":"setting-up-wordpress-and-sql-with-openebs"},{"id":84,"title":"ARMing Kubernetes with OpenEBS \\#1","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"02-08-2018","tags":["Arm64","Kubernetes","Lepotato","Owncloud","Solutions"],"excerpt":"Running stateful containers on Le Potato. Why not! It’s fun and extremely efficient! I know many people (including me) used to run ownCloud on their desktop computers.","content":"\n## Running stateful containers on Le Potato\n\nWhy not! It’s fun and extremely efficient! I know many people (including me) used to run ownCloud on their desktop computers. I have finally decided to retire my old desktop computer and was looking ways to keep my ownCloud instance alive and maybe even improve it a bit.\n\nFirst, I ran **ownCloud** on GKE Kubernetes cluster and came to a decison quickly that it’s not what I needed:\n\n- I am used to the speed of USB 3.0 for large uploads when needed. I wanted to keep the option of (50MB+/sec) using USB. Which means, if I choose a budget ARM SoC route, then the board should have **non-shared bandwidth** for LAN and USB.\n- 4 node GKE cluster using n1-standard-1 + 200GB storage costs ~$225/month, I would rather use Dropbox for $20/month = 240$/year (still doesn’t give me what I need).\n- **Low-power**, possibly solar-powered. I’ll share my power consumption findings in the series of blog articles.\n- Everything on **Kubernetes** is more fun, right?\n\nI was looking into [Raspberry Pi 3 Model B](http://amzn.to/2GMjYt4) option and after a quick trial realized that shared USB/Ethernet bandwidth and lack of MicroSD UHS support is not going to give me the performance I need and found the **AML-S905X-CC Le Potato** board.\n\nLibre Computer Board, code name Le Potato, is designed as a drop in hardware replacement for the Raspberry Pi 3 Model B (In fact it has the exact same form factor and port locations) and offers faster performance, more memory, lower power, higher IO throughput, 4K capabilities, open market components, improved media acceleration and removal of the vendor locked-in interfaces. This platform uses the latest technologies and is built upon proven long-term available chips. It is supported by upstream Linux and has a downstream development package based on Linux 4.9 LTS that offers ready-to-go 4K media decoding, 3D acceleration, and more.\n\nMost importantly, Le Potato has almost double the performance of RPi 3 with 2GB memory and 50% faster CPU and GPU. It also has non-shared bandwidth for LAN and USB and MicroSD UHS support. I was able to get over 70MB/s read&write performance vs ~15–20MB/s on RPi 3. I also noticed that even under heavy load Le Potato has lower power consumption compared to Rpi 3.\n\nIt sounds too good to be true, right? Since Le Potato is new, I’ve decided to run both side-to-side and publish my experience.\n\nIn this blog post, I will focus on setting up a Kubernetes on a Le Potato Clusters, install ownCloud on top, and compare it to Rpi 3.\n\n## Prerequisites\n\n### Hardware\n\n- 4 x [Libre Computer Board AML-S905X-CC (Le Potato) 64-bit (2GB)](http://amzn.to/2ptxGJS) $45\n- 4 x 32GB MicroSD Card ([Samsung 32GB 95MB/s MicroSD](http://amzn.to/2uayTe4) $9.99)\n- 4 x 128GB USB Drive ([Samsung 128GB USB 3.0 Flash Drive Fit (MUF-128BB/AM)](http://amzn.to/2psgFPC) $39)\n- 1 x Desktop Switch ([TP-link 5-Port Gigabit Desktop Switch](http://amzn.to/2u3TCQN) $29.99)\n- 1 x Active USB Hub ([Generic 7-Port USB Hub with ON/OFF Switch](http://amzn.to/2IBIZaO) $5.64)\n- 4 x short USB to Micro USB cable ([ZiBay Micro USB Short Sync Cable for Select Models/Device, 7-Inch — Pack of 5](http://amzn.to/2G8Doub) $6.99)\n- For comparison: 4 x [Raspberry Pi 3 Model B](http://amzn.to/2GMjYt4) $34.62\n\n### Optional:\n\n- Short cables make it look clean and nice:\n 6-inch CAT6 flat network cables ([5-PACK 6-inch CAT6 Network UTP Ethernet RJ45 Flat-Design](http://amzn.to/2GcT5AV) $12.48)\n- One touchscreen to access the cluster when nothing else available:\n 1x 3.5 inch TFT Touch Screen ([kuman 3.5 Inch 480×320 TFT Touch Screen Monitor for Raspberry Pi](http://amzn.to/2pwR9tt) $19.39)\n- I also build a mobile version to run in my car using this with a Tmobile line ([SIM800 Module GSM GPRS Expansion Board UART V2.0](http://amzn.to/2GLBeyE) $25.99)\n\n### Software components used\n\n- [Armbian 5.38 Ubuntu Xenial](https://dl.armbian.com/lepotato/) (for Le Potato)\n- [Raspbian Stretch Lite](https://www.raspberrypi.org/downloads/raspbian/) 2017–11–29 (for Rpi 3)\n- [Etcher](https://etcher.io/) v1.3.1\n- Kubernetes v1.9.2+\n- OpenEBS 0.5.3 arm64\n- ownCloud\n\nI will start with Le Potato and compare against Raspberry Pi 3 on my next blog.\n\n### Flash Le Potato Armbian image on SD Cards\n\n**Armbian** provides Debian and Ubuntu based builds for ARM development boards. Armbian Ubuntu is pretty much same as Ubuntu, except the desktop interface. Armbian uses the Xfce desktop environment, which is a lighter than Gnome or KDE. Its main advantage is its speed, and it’s ideal for systems with 256 MB to 512 MB of RAM. And, I plan to disable desktop anyways.\n\nDownload the [Armbian Ubuntu image](https://dl.armbian.com/lepotato/Ubuntu_xenial_next_desktop.7z) from the link [here](https://dl.armbian.com/lepotato/), extract and burn it to your SD cards using Etcher.\n\n![Etcher flashing](https://cdn-images-1.medium.com/max/800/0*XI8xSg4dCl_IWvbz.png)\n\nPlug the SD card into your Le Potato board and power on.\n\n![Terminal Window](https://cdn-images-1.medium.com/max/800/0*ojAbBScZY7giAV7b.jpg)\n\nLogin as `root` and use password `1234`. You will be prompted to change this password at first login. Next, you will be asked to create a normal user account that is sudo enabled.\n\n### Prepare Armbian host\n\nAfter reboot, you will auto login to your host with the new user you have created.\n\n![Armbian Le Potato](https://cdn-images-1.medium.com/max/800/0*UOLgX8I0Oz9hw7I8.jpg)\n\nChange the hostname and set static IP by using armbian-config utility:\n\n`sudo armbian-config`\n\n![armbian-config utility](https://cdn-images-1.medium.com/max/800/0*s8fjc1-L-9pkDzRE.png)\n\nDisable swap by running the following commands:\n\n```\nsudo systemctl disable zram-configsudo swapoff -a\n```\n\nAnd also comment out the reference to swap in /etc/fstab file:\n\n```\nsudo vi /etc/fstab\n```\n\nAfter reboot, confirm that swap space is disabled by running the following command. It should return empty.\n\n```\nsudo swapon — summary\n```\n\nInstall Golang 1.10:\n\n```\nwget https://dl.google.com/go/go1.10.linux-arm64.tar.gz\nsudo tar -C /usr/local -xzf go1.10.linux-arm64.tar.gz\nexport PATH=$PATH:/usr/local/go/bin\nmkdir go\nexport GOPATH=”$HOME/go”\ngo get github.com/kubernetes-incubator/cri-tools/cmd/crictl\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Install Docker on Armbian Ubuntu (arm64)\n\nRun the following command to install Docker on all nodes. The second line is to use Docker as a non-root user, use your username instead of mine below (murat):\n\n```\ncurl -sL https://get.docker.com | sh\nsudo usermod murat -aG docker\n```\n\nSuccessful installation would look like below:\n\n```\nmurat@kubenode1:~$ curl -sL https://get.docker.com | sh\n # Executing docker install script, commit: 02d7c3c\n + sudo -E sh -c apt-get update -qq >/dev/null\n + sudo -E sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null\n + sudo -E sh -c curl -fsSL “https://download.docker.com/linux/ubuntu/gpg\" | apt-key add -qq →/dev/null\n + sudo -E sh -c echo “deb [arch=arm64] https://download.docker.com/linux/ubuntu xenial edge” > /etc/apt/sources.list.d/docker.list\n + [ ubuntu = debian ]\n + sudo -E sh -c apt-get update -qq >/dev/null\n + sudo -E sh -c apt-get install -y -qq — no-install-recommends docker-ce >/dev/null\n + sudo -E sh -c docker version\nClient:\n Version: 18.02.0-ce\n API version: 1.36\n Go version: go1.9.3\n Git commit: fc4de44\n Built: Wed Feb 7 21:11:48 2018\n OS/Arch: linux/arm64\n Experimental: false\n Orchestrator: swarm\nServer:\n Engine:\n Version: 18.02.0-ce\n API version: 1.36 (minimum version 1.12)\n Go version: go1.9.3\n Git commit: fc4de44\n Built: Wed Feb 7 21:09:57 2018\n OS/Arch: linux/arm64\n Experimental: false\nIf you would like to use Docker as a non-root user, you should now consider\nadding your user to the “docker” group with something like:\nsudo usermod -aG docker murat\nRemember that you will have to log out and back in for this to take effect!\nWARNING: Adding a user to the “docker” group will grant the ability to run\n containers which can be used to obtain root privileges on the\n docker host.\n Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface\n for more information.\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Install Kubernetes on Armbian for Le Potato\n\nRun the following command to install Kubeadm on all nodes:\n\n```\ncurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add — && \\\n echo “deb http://apt.kubernetes.io/ kubernetes-xenial main” | sudo tee /etc/apt/sources.list.d/ kubernetes.list && \\\n sudo apt-get update -q && \\\n sudo apt-get install -qy kubeadm\n```\n\nRepeat all the steps above on all your nodes.\n\n#### Initialize Kubernetes master node\n\nInitialize your master K8s node:\n\n```\nsudo kubeadm init — pod-network-cidr=10.20.0.0/24 — apiserver-advertise-address=10.10.0.131\n```\n\nBy default, token expires in 24h. If you need it longer, then you can add `— token-ttl=0` to the end of the command above to generate token that does not expire.\n\nThis step may take around 10 minutes and after that, you will see a summary like below:\n\n```\n…\nYour Kubernetes master has initialized successfully!\nTo start using your cluster, you need to run the following as a regular user:\nmkdir -p $HOME/.kube\n sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\n sudo chown $(id -u):$(id -g) $HOME/.kube/config\nYou should now deploy a pod network to the cluster.\n Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:\n https://kubernetes.io/docs/concepts/cluster-administration/addons/\nYou can now join any number of machines by running the following on each node\nas root:\nkubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n```\n\nRun the following:\n\n```\nmkdir -p $HOME/.kube\nsudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\nsudo chown $(id -u):$(id -g) $HOME/.kube/config\n```\n\nDeploy a pod network to the cluster. I used flannel, you can see your other options [here](https://kubernetes.io/docs/concepts/cluster-administration/addons/).\n\n```\nsudo sysctl net.bridge.bridge-nf-call-iptables=1\nkubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml\n```\n\nBy default, pods cannot be scheuled on the master node. If you want to be able to schedule pods on the master, e.g. for a single-machine Kubernetes cluster for development, run:\n\n```\nkubectl taint nodes — all node-role.kubernetes.io/master-\n```\n\nAs soon as the pod network has been installed, you can continue by joining your nodes.\n\nTo confirm that kube-dns pod is up run the command below and check the output:\n\n```\nmurat@kubenode1:~$ kubectl get pods — all-namespaces\n NAMESPACE NAME READY STATUS RESTARTS AGE\n kube-system etcd-kubenode1 1/1 Running 0 1m\n kube-system kube-apiserver-kubenode1 1/1 Running 0 1m\n kube-system kube-controller-manager-kubenode1 1/1 Running 0 1m\n kube-system kube-dns-6448b967fc-bc58z 3/3 Running 0 1m\n kube-system kube-proxy-h7p6s 1/1 Running 0 1m\n kube-system kube-scheduler-kubenode1 1/1 Running 0 1m\n [/cce_bash]\n```\n\nNote: If kube-dns is stuck in the Pending state. Follow the steps below to fix it and re init your master. This issue and the solution was mentioned [here](https://github.com/kubernetes/kubernetes/issues/43815).\n\n```\nkubeadm reset\nsudo nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf\n```\n\nRemove the `$KUBELET_NETWORK_ARGS` entry from the ExecStart, save the file, and reload systemd and kube services.\n\n```\nsystemctl daemon-reload\nsystemctl restart kubelet.service\n```\n\nInitialize your master K8s node again.\n\n## Join Kubernetes nodes to the cluster\n\nYou can now join any number of nodes by running the command with the token generated during the K8s master initialization:\n\n```\nmurat@kubenode2:~$ kubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n [preflight] Running pre-flight checks.\n [preflight] Some fatal errors occurred:\n [ERROR IsPrivilegedUser]: user is not running as root\n [preflight] If you know what you are doing, you can make a check non-fatal with ` — ignore-preflight-errors=…`\n murat@kubenode2:~$ sudo kubeadm join — token 17c6f2.bd9fa915e6a2fcfb 10.10.0.131:6443 — discovery-token-ca-cert-hash sha256:b4995d14fc8995d5ac271e49772b1cf5aa9fee48fa2729fd4ca7fefbbb0564ac\n [preflight] Running pre-flight checks.\n [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.0-ce. Max validated version: 17.03\n [discovery] Trying to connect to API Server “10.10.0.131:6443”\n [discovery] Created cluster-info discovery client, requesting info from “https://10.10.0.131:6443\"\n [discovery] Requesting info from “https://10.10.0.131:6443\" again to validate TLS against the pinned public key\n [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server “10.10.0.131:6443”\n [discovery] Successfully established connection with API Server “10.10.0.131:6443”\nThis node has joined the cluster:\n * Certificate signing request was sent to master and a response\n was received.\n * The Kubelet was informed of the new secure connection details.\n```\n\nRun `kubectl get nodes` on the master to see this node join the cluster.\n\nIf you forgot the cluster token, you can generate a new one with the command:\n\n```\nkubeadm token generate\n```\n\nRepeat all the steps above on all your nodes.\n\n## Install OpenEBS on ARM (Le Potato)\n\nSimilar to most of the arm based hobby boards, Le Potato doesn’t provide any additional redundancy. Even using a RAID protected external USB device wouldn’t give me protection against node failure unless it’s some form of a shared network storage. They are both way over my affordability requirement. All I need is a replicated block device, so my container can survive a node or USB device failures.\n\nOpenEBS provides a great solution for modern x64 architecture but currently doesn’t have a build for [arm64](https://en.wikipedia.org/wiki/ARM_architecture#64/32-bit_architecture) (armv8) architecture. Therefore, I’ve opened an issue [here](https://github.com/openebs/openebs/issues/1295) and started working on it myself. I did successfully build OpenEBS images for arm64 architecture from the repo base on the 0.5.3 release and uploaded custom images to my personal docker registry [here](https://hub.docker.com/u/muratkarslioglu/). So, it is work in progress and please use it at your own risk, until it’s merged.\n\n```\nsudo apt-get install -y curl open-iscsi\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/openebs-operator-arm64.yaml\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/openebs-storageclasses.yaml\n```\n\nNow, get the list of storage classes using the below command:\n\n```\n$ kubectl get sc\n NAME PROVISIONER AGE\n openebs-cassandra openebs.io/provisioner-iscsi 1h\n openebs-es-data-sc openebs.io/provisioner-iscsi 1h\n openebs-jupyter openebs.io/provisioner-iscsi 1h\n openebs-kafka openebs.io/provisioner-iscsi 1h\n openebs-mongodb openebs.io/provisioner-iscsi 1h\n openebs-percona openebs.io/provisioner-iscsi 1h\n openebs-redis openebs.io/provisioner-iscsi 1h\n openebs-standalone openebs.io/provisioner-iscsi 1h\n openebs-standard openebs.io/provisioner-iscsi 4d\n openebs-zk openebs.io/provisioner-iscsi 1h\n```\n\n**Voila…!**\n\n`openebs-standard` storage class creates 3 replicas. That’s what I will use for my application.\n\nTo test the OpenEBS installation you can try my Jenkins example here:\n\n```\nkubectl apply -f https://raw.githubusercontent.com/muratkars/openebs/lepotato-arm64/k8s/demo/jenkins/jenkins-arm64.yaml\n```\n\n## Next — Installing containerized OwnCloud on OpenEBS\n\nFinding right container images to run on arm64 architecture is challenging. On the next article, I will build an OwnCloud container image running on Postgres database and both containers will store their data on OpenEBS persistent volumes.\n\nMy final goal is to build a mobile OwnCloud cluster installed in my family van, where storage is replicated to another cluster in my home lab.\n\nStay tuned!\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/arming-kubernetes-with-openebs-1/)_._\n","notHasFeatureImage":false,"slug":"arming-kubernetes-with-openebs-\\#1"},{"id":85,"title":"Recap of Google Next’18","author":"Murat Karslioglu","author_info":"No author information","date":"01-08-2018","tags":["Cloud Services Platform","Knative","Google Next18","Istio","OpenEBS"],"excerpt":"This year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.","content":"\n![You know that you are at the right event when you see a familiar face like Mr. Hightower :)](https://lh3.googleusercontent.com/iBQD9nOCN5cmrzn73zLeMoHDdhbTZWa3d4sSC1k1wkudXXL0L0912hrjUe2Bxr3MBTLOM_-LDC-ZrA-zNq8arcTJfD_V6e0pc_A9_oKcm6tAsBnIfqXdTfEbnmb8Qu_PoSyBZkVN)\n\nThis year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.\n\nNext ’18 was an excellent representation of Google as a company and their culture. Sessions were mostly in Moscone West, but the whole event was spread across Moscone West, the brand new South building, and six other buildings.\n\n![Next ’18 Event Map](https://lh4.googleusercontent.com/3ojmOPqqjEieE6GxfEjgFxRRv4sIzQpA_x21hFRpj3IRrmy6i7HL4k5FO2zztbwf9b5HJlrzO8BP3bWkOM34gZQdKS5lmLqR0FjmHJr96VIToFfc-SWdIKmlLcMJLz2y_tWPbERn)\n\nThe floor plan was fun and casual; catering was of “Google Quality” and the security was insane, with metal detectors, police, K9 search dogs, and cameras everywhere. And of course games, fun, and even “Chrome Enterprise Grab n Go” were there in case you needed a loaner laptop to work on — see some pictures at the end. :)\n\n### **What I learned at the Next ’18 conference**\n\nFirst of all, a big shout out to all involved in the [**Istio** project](https://istio.io/). It is not a surprise that we see great advocate marketing and support for the Istio 1.0 GA release on social media last week. Istio is a big part of the [**Google’s Cloud Services Platform**](https://cloud.google.com/solutions/cloud-services-platform/)(**CSP**) puzzle.\n\n![GCSP Dashboard — After deploying my first app in less than 30 seconds.](https://lh4.googleusercontent.com/taoSgNELqkMCnfsqUd84nPfbIATkjucboLYdbzMUWKct5ZFiXb_PZFjoU5KFBc5LZNBz6mhuwHXEpMs49tREabCzrMDCuNTrKniQ4UYuk_i1pNxR08pUHOEjtZ3nxfUOZswcA_xe)\n\nLater this year, Google is targeting to make all components of their CSP available (in some form). CSP will combine **Kubernetes**, **GKE**, **GKE On-Prem** and **Istio** with Google’s infrastructure, security, and operations to increase velocity, reliability, and manage governance at scale.\n\nCloud Services Platform will be extensible through an open ecosystem. **Stackdriver Monitoring** and **Marketplace** are the extensions to platform services. [Marketplace](https://console.cloud.google.com/marketplace/browse?filter=solution-type:k8s) already has 27 Kubernetes apps including commonly used components of many environments such as Elasticsearch and Cassandra.\n\n![CSP Marketplace](https://lh3.googleusercontent.com/esTW1l0iBV-Wvleoosxha1W5KqmA5BQLZ-4jyfb3e0W2j_S5rzqtncJCFA8t6brQc_ZJdF2eVqaXAdhHASBlTq9izYO85SLSZRyE8mbwoB1EiFHTmQdwDHsnTdFm2EDb0i4yefVA)\n\nUsers will be able to deploy a unified architecture, that spans from their private cloud, using Google CSP to Google’s public cloud. Again, the two most important pieces to this puzzle are managed versions of the open source projects Kubernetes and Istio. To me, the rest of it still feels mostly to be DIY-like quality.\n\nKnative, Cloud Build, and CD are other significant solutions announced at Next’18.\n\n### **A new cloud availability zone, this time in your datacenter — which might be in your garage**\n\nAt first, **GKE on-prem** got me interested. But, after talking to a few Google Cloud Experts again, I felt it’s very early to be seriously considered. You can read others’ thoughts here on [Hacker News](https://news.ycombinator.com/item?id=17602555).\n\n![Discussions on GKE on-prem](https://lh5.googleusercontent.com/q_0yHazRpRZ9bLptYCw1_GQsmdM8TpbM7xXmZ8nL8nejR3uhVg79bvJokA_BrQ91VfzYYl8OLtQ1Evl5VNJqDwJM3EKwqKFsn_jr99N91hCEa1lkazjMZE3aphRbr21LEc0atqTr)\n\nGKE on-prem alpha will support **_vSphere 6.5_** only, no bare-metal for now!\n\nFailover from on-prem -> GKE is something Google team is working on. This means GKE on-prem instance will look like another availability zone (AZ) on a Google Cloud dashboard.\n\nOther than vSphere dependency, the idea of being able to have an availability zone, local in your data center is really compelling. It is also a very common use-case for [**OpenEBS**](https://openebs.io/) since there is no cloud vendor provided, a cloud-native way of spreading your cloud volumes, EBS, etc. across AZs — we see many community users running web services today using OpenEBS to enable that.\n\n### **Github and Google Partnership to provide a CI/CD platform**\n\n**Cloud Build** is Google’s fully managed CI/CD platform that lets you build and test applications in the cloud. Cloud Build is fully integrated with GitHub workflow, simplifies CI processes on top of your GitHub repositories.\n\n![Me deploying myself on Serverless Cloud Maker :)](https://lh3.googleusercontent.com/Vid2Mpm0eaSATtriAt3eLoDdBvvRcv7WCJeNBKxe_VOhVcbdrmh_nJIn5aiQlnMfEOpywRMhHF7Gnv58Nyu_5MQHoWWfxMCmPYdfDlYlKkiQPldJvHxEk9Qa5BOQBuDQNW-YZ0dc)\n\nCloud Build features;\n\n**Multiple environment support** lets developers build, test, and deploy across multiple environments such as VMs, serverless, Kubernetes, or Firebase.\n\n**Native Docker support** means that deployment to Kubernetes or GKE can be automated by just importing your Docker files.\n\n**Generous free tier**— 20 free build-minutes per day and up to 10 concurrent builds may be good enough for many small projects.\n\n**Vulnerability identification** performs built-in package vulnerability scanning for Ubuntu, Debian, and Alpine container images.\n\n**Build locally or in the cloud** enables more edge usage or GKE on-prem.\n\n### Serverless — here we are again\n\n**Knative** is a new open-source project started by engineers from Google, Pivotal, IBM, and a few others. It’s a K8s-based platform to build, deploy, and manage serverless workloads.\n\n_“The biggest concern on Knative is the dependency on Istio.”_\n\nTraffic management is critical for serverless workloads. Knative is tied to Istio and can’t take advantage of the broad ecosystem. This means existing external DNS services and cert-managers cannot be used. I believe, Knative still needs some work and not ready for prime-time. If you don’t believe me, read the installation YAML file — I mean the 17K lines “human readable” configuration file ([release.yaml](https://github.com/knative/serving/releases/download/v0.1.0/release.yaml)).\n\n![](https://lh6.googleusercontent.com/0qn1GCe8B-15DIr5G7eqqbg3FfnOcm58iQ08ZUobrKJ82xIArtNjnSuFS2KOkkEhyGfyTH8pz5_NXZOk87EllIjN4rSVYlyxxmN6iDemZ0AgM_Yd-FMZzMR-nQdCHpFPTIL84hwS)\n\n### **My take on all of the above — Clash of the Cloud Vendors**\n\nIf you have been in IT long enough, you could easily see the pattern and predict why some technologies will become more important and why will the others be replaced.\n\n_“What is happening today in the industry is the battle to become the “Top-level API” vendor.”_\n\n20–25 years ago hardware was still the king of IT. Brand-name server, network, and storage appliance vendors were ruling in the datacenters. Being able to manage network routers or configure proprietary storage appliances were the most wanted skills. We were talking to hardware…\n\n20 years ago (in 1998), VMware was founded. VMware slowly but successfully commercialized hypervisors and virtualized the IT. They became the new API to talk to, everything else under that layer became a commodity. We were suddenly writing virtualized drivers, talking software-defined storage and networking — the term “software-defined” was born. Traditional hardware vendors lost the market and momentum!\n\n12 years ago, the AWS platform was launched. Cloud vendors became the new API that developers wanted to talk to, hypervisors became a commodity. CIO and enterprises that are sucked into the cloud started worrying about the cloud lock-in. Just like the vendor lock-in or hypervisor lock-in, we have experienced before. Technology might be new, but concerns were almost the same.\n\n4 years ago, Kubernetes was announced and v1.0 released in mid-2015. Finally, an open-source project that threatens all previous, proprietary, vendor managed “Top-level API” that we were using became a majorly adopted container orchestration technology. Although it came from Google, it took off after it got open-sourced and probably would be right to say that so far financially, Red Hat profited most from Kubernetes with their Red Hat OpenShift platform. And now we see somewhat of a battle over APIs to be used in operating applications on Kubernetes, with the RedHat / CoreOS operator framework and other projects including one supported by Google and others such as Rook.io emerging to challenge or extend the framework.\n\nGoogle Container Engine (**GKE**), Microsoft Azure Container Service (AKS**)**, Amazon Elastic Container Service (**EKS**), IBM Cloud Container Service (**CCS**), and Rackspace Kubernetes-as-a-Service (**KaaS**) are all competing in the hosted Kubernetes space (new vendors expected here).\n\nThere is enough space to grow in the self-hosted Kubernetes space. GKE on-prem is the validation from Google.\n\n**Hardware>Virtualization>Cloud>Containers>Serverless???**\n\nMany of us see **Serverless** as the next step, but it might be too granular to support larger adoption and current limitations validate the claims. It doesn’t scale well for intense workloads.\n\nOne size doesn’t fit all, there are still traditional use cases that even run on bare-metal and VMs. Same might be true for Serverless. It is not for every workload. Modernizing existing workloads will take time, and we will see who will become the leader of the next “Top-level API”.\n\nWhat do you think? Who is going to win the clash of the titans? What did you think about Next’18 and Google’s strategy?\n\nThanks for reading and for any feedback.\n\n### **Some Next’18 moments from my camera**\n\n![](https://lh4.googleusercontent.com/ARfwggxkEIm1I-QXUGinQGV0zVQLzaTaQ9WxUEC4nN-xuTUsK0I-Bi4JO9kwyIi6MQYxnu0hBQDxdbkVy5nsTd5oQMEl-JCXRvdWWVhcrbCK3EfM8EegXImT2_Kn0kXeZoHbfLmK)\n\n![](https://lh5.googleusercontent.com/bUCoD0IKci0QEETmHlcrUN-wOSLFSYsIuR4aG96D3QtZo23_gm10SfBqMzUtHVduFt-XzA4m9mI-sae4ktxJRIG9m9aBUt9VUtG0ytYyjFoh-Q2GbFNlQC7Ry0iBiTiaKUNuKsFd)\n\n![](https://lh3.googleusercontent.com/Vdr19dKixjLs64xGxB_thJ3D8_-cnMihbC-gH50S5FuFJ13y2UMb42zSQy9Rp2LT9olLP5TPSRb4WPoW71l71NMJoBeK70SiFtYgsDs1l2k2tmLTiqJTlo9ajj0F0xTp4JlkQuF2)\n\n![](https://lh4.googleusercontent.com/GGkcm_nFqb_gDKJuBJISMI3mueZx6xiBE6R8diM84xEOnYmcSDQVNPaRTbIFgBf-fh1Y_8JcioxxP8g8RzrxYEUJ4Xhw51RMRdXw3aS3fVpbH_mjc_kPUC0pXL3WxXZNlgR3baCb)\n\n![](https://lh4.googleusercontent.com/seBUEu7ltrIHW3VxY_V5rqekHkLXfLYhR0dZZbsyr8MC-vww-_rGvQA3rPz0kfbqudV-LNAi292BTwNrGIe_KG56Vqr-sHSv-mPlIy8nhnJDQPQKnUj7ohg3uql67RVvqTf-earr)\n\n![](https://lh6.googleusercontent.com/2BPdWJYXZizBdmFra0GYqedeArDGynas8VsxPkC0FWhyCjUm8A7TYLpPN06hPMmQ1UQ_yPoG8mH91eRjAyQE6sbw4Jo2wjPlsqDVTLEohowMOtSQaEuSWZO0lIntDOeMip75K20P)\n\n![](https://lh3.googleusercontent.com/esI-PdVDc9sEqPoOkwG4nQhcD_FYSxs8Z1eBiHW9UTBDNO9bbd145X9vwnQgijXHTiz6DUD_bgkz9ViC1C2ukDYtjLaHVAIFlEMPcHmQEWjpKeq3pvEy4HyWXeK65oe61LUMTZ1-)\n\n![](https://lh5.googleusercontent.com/KtS0m1tRBvjWehJSCSFItgtvDK5IiAEU20aa3GfSK6TmlyPVWjQpjnq_z5OAxsa1-L7PQuNRuiK2ZRX1It-1CrDlqzv1ubwrYaZA_gQQGxsb2rXJCAiQPjZ9GLiqXHZsCess6dYz)\n\n![](https://lh6.googleusercontent.com/Uee-fb3QJewe05s4AWM2bF6b7MuRI8XgU9r3KfX72RNwVYYecjt5UX15vw1jhk0LoqvgL1MN5yKT5t9Mei6QzI7bohUAKtoQthG02YCr1VXiM4HFB-RRmQB29uuANQNDiq1sGKGe)\n\n![](https://lh6.googleusercontent.com/OXx7ZGSWDI5z2UnqunkcWtB1MWY5ZtXs3EmruVAqfZo4JiLzhd00hmRkKZi_y2Icv6FV4CJRJW68HK1laKNXCKnGI5A9Z1l53R1BtOiM6dLzjDvecuWLgzPIgir3Q89qxkHt50yo)\n\n![](https://lh4.googleusercontent.com/LUlB3APRP1t0gEEzaXmPbUxZFKGD1nVHRtdNMoKp7iSTLAZAHXOF3W_VPaZs9-XsLdw54GC2TnjnxGyR_spqek8X5ZLFkWICZpP4oXEzATj4n_vfvLQtr0FNceiT3lKzGKg7oh1f)\n\n![](https://lh4.googleusercontent.com/AZP7UsS_i0gXvSBRuoDas6D48cBdU9JD3N39C53YEIdJwXARCyEg2sXRSuvOGUSa07o0xZ2UzQc8mLgZxj6EQm8uRLZ3JXVDzEwx6kTT3Vq-eia5OV865zc0q3bq3htEDZxUiIX1)\n\n![](https://lh4.googleusercontent.com/GkXUs0HYRwS66hSMex7NDDT6Ck0NaxO4VrIHW23GzjrQbRMWDDA4EGwIOwSsg3O8tL_iNuISRywMDuF1u61rZm4fzaBisfUkyo-aPIcCcTk3KwUPunnPwgrV6-oRca0td-eYHxss)\n\n![](https://lh3.googleusercontent.com/EPOoqoV_b11WJCV9cNJFgAtGfuRyqF3xZpKKoJ46J54wNsrE0kFFp82WFZn_gRoRTSLYPAkFHdFcizdOOBYwk_pL6pUZiy9ld24o-xBHIJZWcLmz_tMSFav77_9fbc2pPbwgnObR)\n\n![](https://lh5.googleusercontent.com/cbuvmjdXzUEgiSf6BL6jC3-TKAbwf1WASiNfypDXQcmeXLrr6OPhITsEjeEUo1dXtg3OW3YTPNez3XxjaRxeAV_Ox6dTq9XpbwNDqh2-iWiXqWhrgn2o1VIe45xKPGYsxxPluLkw)\n\n![](https://lh5.googleusercontent.com/wskI0Jy2g9icXoitcNDIp9VChhWlxu9fIDwepnqd0ds_Oq1m8Yt7ZHY8GZ_DvfS8IQcNfsJyBanCPIpp_GUznnzK3b3YBP7F2oV9gsi9k9WyKfwmWGCI4um2SKeNveXy2uaeD-sd)\n\n![](https://lh5.googleusercontent.com/zeDLw8eZvkjgSfWnUwOkIG_Ze3GmKAwrTH59o37K8XEMXDMcAOtUsOSzfEwP8qF7qgVHEbgCs6YbXyE9loDTZOX4q5JeRe2J4JMMufrD0H6wr-ADk9sMBzRRwgi3iRVjhTCLh0Uy)\n\n### **Check-out the popular hashtags :**\n\n[#next18](https://twitter.com/search?q=next18) [#googlenext18](https://twitter.com/search?q=googlenext18) [#knative](https://twitter.com/search?q=knative) [#kubernetes](https://twitter.com/search?q=kubernetes) [#istiomesh](https://twitter.com/search?q=istiomesh)\n\nAlso check out the keynotes:Keynotes from last week’s[ #GoogleNext18](https://twitter.com/hashtag/GoogleNext18?src=hash) here →[ http://g.co/nextonair](https://t.co/mVuwk0hw4i)\n","notHasFeatureImage":false,"slug":"recap-of-google-next’18"},{"id":86,"title":"Designating OpenEBS pods to run on specific K8S Nodes","author":"Ajesh Baby","author_info":"Product Manager at MayaData","date":"30-07-2018","tags":["OpenEBS","Kubernetes","Solutions","Scheduler","Scheduling"],"excerpt":"OpenEBS does not have a separate scheduler used to manage scheduling pods. Instead, it uses the Kubernetes scheduler for managing the scheduling needs of an administrator.","content":"\nOpenEBS does not have a separate scheduler used to manage scheduling pods. Instead, it uses the Kubernetes scheduler for managing the scheduling needs of an administrator. Kubernetes provides the following methods for controlling the scheduling pods on cluster nodes:\n\n- [nodeSelector](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector)\n- [Taints and Tolerations](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)\n\nFor more details and information about these features, you can refer to the Kubernetes documentation.\n\nIn this post, I would like to cover the different aspects of how to restrict/control the OpenEBS pods scheduling to a set of specific nodes in the Kubernetes cluster.\n\nOpenEBS deals with many types of Kubernetes pods throughout its life cycle. These can be broadly categorized into two types, control plane pods and data plane pods. Control plane pods are installed as part of installation of the following OpenEBS components:\n\n- OpeEBS API Server\n- OpenEBS Provisioner\n- OpenEBS Snapshot Controller\n\nData plane pods are installed as part of volume provisioning:\n\n- Target Pod\n- Replica Pods\n\nFor details on the exact steps of scheduling, see the configuration section [here](https://docs.openebs.io/docs/next/scheduler.html?__hstc=216392137.e7b2938c542eaf0f98426e5d8be4aa84.1579859056424.1579859056424.1579859056424.1&__hssc=216392137.1.1579859056424&__hsfp=3765904294).\n\nUse case: Let’s consider a scenario in which you have 20 nodes named Node1, Node2 ... Node20. You may want to designate Node1, Node2, Node3 as storage nodes so that all storage pods are scheduled only on these nodes.\n\nSolution: You can use Kubernetes scheduling methods to achieve this. Below are some of the possible options and their effect on scheduling pods to respective nodes.\n\n![Storage Pods](/images/blog/storage-pods.png)\n\nYou may select and use any of the above options based on your unique requirements.\n\nOption 2 does not necessarily guarantee storage pod scheduling on Node1, Node2 and Node3.\n\nOption 1, Option 3, and Option 4 will limit the scheduled OpenEBS pods to Node1, Node2 and Node3. Option 3 is my preferred choice, for the following reasons:\n\n- Other application pods will not be scheduled on my storage nodes, whereas Option 1 does present the possibility of other application pods being scheduled on Node1, Node2, and Node3.\n- While scaling the cluster for application deployments, I do not have to worry about changing the policy for storage. If I use option4, I must taint the new nodes with respective applications.\n- In this scenario, I am worried only about storage nodes, as these have local disks attached to them. I am not restricted to schedule an application pod deployment on any nodes other than storage nodes.\n","notHasFeatureImage":false,"slug":"designating-openebs-pods-to-run-on-specific-k8s-nodes"},{"id":87,"title":"OpenEBS 0.6 serves IOs amidst Chaos and much more","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"20-07-2018","tags":["DevOps","Kubernetes","OpenEBS","Stateful Containers","Storage"],"excerpt":"We are very excited to announce the availability of OpenEBS version 0.6. I would like to take a few minutes to talk about what has been keeping us busy in making it GA.","content":"\nWe are very excited to announce the availability of OpenEBS version 0.6. I would like to take a few minutes to talk about what has been keeping us busy in making it GA. We have been making a number of additions to both the design and code in the last few months. In this blog I’ll talk about:\n\n- Quality leveraging Chaos Engineering\n- Framework (CAS Templates) for supporting multiple storage engines\n\nI won’t talk about our new Storage Engine (cStor) — written in C — which is _almost_ ready. I’ll save that for a later blog.\n\nBefore going into specifics, I would like to express my sincerest gratitude to the OpenEBS community users and developers who are helping to make OpenEBS the most simple and easy to setup containerized attached storage (CAS) solution available for Kubernetes today — and the most popular open source one as well.\n\nSince the OpenEBS 0.5 release we have seen so many ways users have deployed OpenEBS many of which we had not envisioned when we started OpenEBS back in 2016. We are working hard to listen to the growing user community. Along these lines, we have a survey that is running through the end of July that takes 2–3 minutes that has proven helpful; please do fill it out if you have not already and we will even send you some swag: [https://www.surveymonkey.com/r/BRDCCWY](https://www.surveymonkey.com/r/BRDCCWY)\n\nThe timing of OpenEBS 0.5 was perfect in that it coincided with a take-off in interest in stateful workloads on Kubernetes. Some deployment patterns I’ve encountered just in the last few weeks include:\n\n- GitLab for internal IT teams\n- Kube weekly just featured a step by step blog on this subject: [https://openebs.io/blog/git-freedom-on-kubernetes](/blog/git-freedom-on-kubernetes)\n- Data Science training sessions\n- Here we are seeing hundreds of pods with stateful workloads spun up and destroyed repeatedly — really a great use case for container attached storage\n- Running Minio on OpenEBS which some users have called a happy marriage between OpenEBS block and Minio S3\n- We remain huge fans of Minio and are looking forward to more community led collaboration with our almost neighbors\n- OpenEBS being deployed as a basic part of DBs on Kubernetes; in particular we are seeing a good amount of NuoDB on OpenShift for example\n- The elastic SQL technology of NuoDB seems to resonate with lots and lots of users; we’re pretty proud that using OpenEBS underneath is becoming a common pattern\n- And of course many Containers as a Service offerings now include OpenEBS as a default option with more to be announced shortly\n\nAnd all this adoption means heterogeneity and dynamism.\n\n## Challenge 1: Kubernetes is resilient amidst Chaos and so must be storage\n\nBecause OpenEBS is deployed on so many varieties of Kubernetes and our fundamental job is to keep the data safe no matter what — we have been investing heavily in our ability to create these varied environments and their behaviors and to then measure and validate the resilience of OpenEBS as these environments respond to outages and increased load and so forth. We are seeing OpenEBS deployed across lots of varieties including:\n\n- Native Kubernetes or using Rancher, OpenShift, IBM Cloud Private, GKE, Tectonic, StackPoint Cloud, and others\n- Operating Systems -Ubuntu, CentOS, CoreOS and others\n- Pod Networking of all types, with Flannel being a favorite\n- Various cloud services — AWS EC2 remains the preferred option, with GCE growing in adoption amongst OpenEBS users\n\nEach combination comes with nuances that are unique and sometimes annoying as well. For example, recently a user on a Cloud Provider saw their nodes shut down frequently and occasional high network latency or packet drops in inter pod networking. Anyone with experience working with Storage Systems knows how detrimental these situations can be for latency-sensitive Stateful Applications.\n\nWe consciously chose the well-understood and widely used iSCSI protocol as the underlying storage connectivity used by Applications to connect to OpenEBS Volumes. There are many benefits to this architecture, but I will not address those here.\n\nThere are some annoying pieces when running iSCSI at scale as well. For example, depending on the response from the iSCSI targets and your operating system, there are some quirky things that can happen with iSCSI. The most notorious of these happens when iSCSI backed volumes move into read-only if you are using ext4 under certain conditions. You must then go through the steps for manually recovering the volumes. To address this, we have put together a troubleshooting guide that you can access [here](/docs/troubleshooting/volume-provisioning?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294).\n\nHowever, we wanted to solve as many of these issues as possible with the right approach. We stepped up our use of chaos engineering in our OpenEBS development process. We also extended and open sourced our in-house tooling, and we are starting to see it used more and more by engineers deploying stateful workloads on Kubernetes — whether or not OpenEBS is the underlying storage.\n\n## Solution : Chaos Engineered OpenEBS, the birth of Litmus.\n\nIf you would like an introduction to the Litmus project, which we open sourced at KubeCon in Denmark, visit the following link: [https://github.com/openebs/litmus](https://github.com/openebs/litmus)\n\nWe are also working on operators to add additional autonomic function into OpenEBS, leveraging improved metrics and advancements in CSI around node daemonsets and the mount-propagation feature. In the meantime, we use Litmus to increase automated real-world scenario testing to ensure improvements in every release. In this regard, a lot of effort has gone towards beefing up the tests that can simulate Chaos at Node, Network, Storage, RAM, and CPU. These typically contribute to Volume Pods switching nodes and, if not careful, interrupted IOs.\n\nOf course, this Chaos for Storage Application is something we believe should be applied to stateful workloads and underlying storage both during testing and as a part of a healthy chaos engineering practice. This is what led us to Open Source Litmus.\n\nOne outcome of our improved chaos engineering and testing is improvements to the resiliency of intra OpenEBS deployment communication. Specifically, we added enhancements to the responses sent by the iSCSI Target to the initiator; overall, this makes OpenEBS more resilient even when Pods are rescheduled unexpectedly and when the environment otherwise changes. You can learn more about these issues in the [release notes](https://github.com/openebs/openebs/releases/tag/v0.6).\n\nWe expect that the incidence of read-only issues will decrease greatly for the tested scenarios. We are constantly adding more scenarios, workloads and other tooling into Litmus to bolster the Jiva storage engine and other engines to come. Contributions to Jiva are of course always welcome!\n\n## Challenge 2: The evolving state of the State in Kubernetes!\n\nIf you regularly monitor storage developments, you will notice that Kubernetes is moving towards CSI and Snapshots are beginning to become a standard. There are enhancements to support Block Volumes and Topology aware Scheduling for Stateful Applications powered by Local PVs, which also benefits other PV types like OpenEBS.\n\nTo give an example, OpenEBS strongly prefers the case when the OpenEBS Controller Pod (and the application Pod) schedule on nodes where the OpenEBS Replica Pod resides. Currently, we achieve this via Pod/Node Affinity parameters. However, with Topology Aware scheduling, the constructs of pinning are efficiently done via the PV topology parameters.\n\nThat is just one example of new capabilities that we must now embrace. Features in Kubernetes now transition “quickly” from alpha to beta and the new paradigms/patterns that enter into Kubernetes must be adopted, or you will soon become outdated like the Third Party Resources (TPRs). However, we are not complaining about the pace of progress and continually contribute upstream to Kubernetes itself. We always seek to lend a hand to make Kubernetes an even better platform for storage and stateful workloads.\n\nNonetheless, the challenge remains. After all, the core of the Orchestration layer in OpenEBS is to deploy and operate the Container Attached Storage solution using Kubernetes native constructs. And the constructs just keep changing!\n\nLooking at the situation, we decided to step back and think about an architecture that would allow us to minimize the need to make code changes every time Kubernetes changes. For example, some users want to deploy their OpenEBS by specifying Pod Disruption Budges (PDBs) or setting specific Resource Limits for PVs in certain namespaces/users etc. We wondered: how can we embrace these new knobs, settings, and advances without endless code churn? This type of work — effectively upgrading the transmission of the underlying orchestration of OpenEBS — is not easy to do unless you really understand the architecture of OpenEBS. That’s not good — what’s the point of being open source if the code itself is too hard to work with and adjust? Fortunately, Kubernetes has CRDs, which provides a way forward.\n\n## Solution : Provide templates to Cluster Owners to define and manage the storage infrastructure.\n\nIn the OpenEBS 0.6 release, we have utilized the power of Kubernetes CRDs to provide a workable solution to introduce pluggable storage engines. OpenEBS now provides a complete workflow for developers and cluster administrators to choose the right storage software and hardware for their unique requirements. The control of the storage infrastructure stays with the cluster owner, and the ability to address a given need in storage lies with the developer. OpenEBS 0.6 brings the initial version of CAS Templates, which are YAMLs that can be scripted by cluster owners, fit into your GitOps, and are associated with Storage Classes.\n\nWe like the way OpenEBS CAS Templates are shaping up, and we can see many of the cluster owners’ needs being met over time, including enforcing of policies using tools like OPA. I will share more on this in upcoming blogs, but you can glance at them by reading this introductory documentation [here](/docs/concepts/cstor?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294). We intend to build upon this improved architectural pattern to do even more than the pluggability of storage engines. As always, we would especially welcome your feedback and use cases.\n\n### And there is more…\n\nYou will notice when you look at the release notes or try OpenEBS 0.6 that there are many other enhancements, including:\n\n- Configuring of OpenEBS for running stateful workloads that span across Availability Zones\n- Enabling the management of Snapshots and Clone from kubectl\n- Enhancement to mayactl to display volume status\n- Improved Integration and Unit Testing coverage\n- Enhanced Contributor Guides\n\nAnd, most importantly, product documentation has been overhauled to provide accessible insights about OpenEBS as well as a process to provide feedback.\n\nAs mentioned above, our next release also enables users to try out _cStor — a storage engine_ that is more efficient in terms of performance and capacity management. It also reduces the number of containers required to run OpenEBS. If you are interested in taking a look, please get in touch as we have some alpha users of cStor now.\n\nWith its strong community of users, developers, and partners building us into their solutions, it feels like OpenEBS is nearly unstoppable. As always, we look forward to your feedback and suggestions on this release and the direction that you want to see OpenEBS move going foward. Please reach out to us on Slack or add comments below. [https://slack.openebs.io](https://slack.openebs.io/?__hstc=216392137.386b1bc3a48de21192b74b07a4e27366.1580120418429.1580120418429.1580120418429.1&__hssc=216392137.1.1580120418429&__hsfp=3765904294)/\n\n[Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":false,"slug":"openebs-0.6-serves-ios-amidst-chaos-and-much-more"},{"id":88,"title":"How do I run a litmus test to compare storage performance on Kubernetes","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"16-07-2018","tags":["Benchmarking","Kubernetes","Litmus","OpenEBS","Solutions","Chaos Engineering"],"excerpt":"Ever so often, developers and devops engineers building or managing stateful applications on Kubernetes are on the lookout for for suitable storage options which serves their application’s specific needs.","content":"\nThis article belongs to a #HowDoI series on Kubernetes and Litmus\n\nEver so often, developers and devops engineers building or managing stateful applications on Kubernetes are on the lookout for for suitable storage options which serves their application’s specific needs. The emphasis could be on high-availability, provisioning ease, performance etc.., **Litmus** (as detailed in this [article](/blog/litmus-release-a-chaos-monkey-on-your-kubernetes-stateful-workloads)), is an attempt to arm them with the necessary info to make the right choice. One of the important storage tests is to simulate application workloads or multiply its effect using synthetic workload generators like fio. In this article, we list the steps to run a fio-based benchmark test using litmus\n\n![Evaluating storage performance w/ Litmus](https://cdn-images-1.medium.com/max/800/1*zRIZ9WjL7S0wq6Sp_IbzCw.png)\n\n## PRE-REQUISITES\n\n- At least a single-node Kubernetes cluster with the necessary disk resources, mounted on the node. (**_Note_**: _Certain storage solutions need minimum Kubernetes versions from which they are supported. For ex: Local PVs are beta from 1.10, OpenEBS needs 1.7.5+_)\n- Storage operator installed (typically, this includes control-plane elements like the static/dynamic provisioners, storage classes and other elements) with appropriate references to the node & disk resources (**_For example_**: _This may involve storage pool creation OR updating disk and node details in the static provisioners etc.,_)\n\n## STEP-1: Setup Litmus essentials on the Kubernetes cluster\n\n- Obtain the Litmus Git repository via a Git Clone operation on the Kubernetes master/Control machine used to manage cluster & set up the Litmus namespace, service account & clusterrolebinding by applying _rbac.yaml_\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ git clone https://github.com/openebs/litmus.git\n\nCloning into 'litmus'...\n\nremote: Counting objects: 2627, done.\n\nremote: Compressing objects: 100% (16/16), done.\n\nremote: Total 2627 (delta 2), reused 9 (delta 2), pack-reused 2609\n\nReceiving objects: 100% (2627/2627), 10.50 MiB | 4.23 MiB/s, done.\n\nResolving deltas: 100% (740/740), done.\n\nkarthik_s@cloudshell:~ (strong-eon-153112)$ cd litmus/\n\nkarthik_s@cloudshell:~/litmus (strong-eon-153112)$ kubectl apply -f hack/rbac.yaml\n\nnamespace \"litmus\" created\n\nserviceaccount \"litmus\" created\n\nclusterrole \"litmus\" created\n\nclusterrolebinding \"litmus\" created\n```\n\n- Create a configmap resource out of the cluster’s config file, typically at _~/.kube/config_, _/etc/kubernetes/admin.conf_ or elsewhere depending on the type of cluster or setup method\n\n(**Note**: _Copy the config file to admin.conf before creating the configmap out of it, as the litmus job expects this path_)\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ kubectl create configmap kubeconfig --from-file=admin.conf -n litmus\n\nconfigmap \"kubeconfig\" created\n```\n\n## STEP-2: Update the Litmus test job as per need\n\nThe litmus fio test job allows the developer to specify certain test parameters via ENV variables, such as the following:\n\n- The litmus fio test job allows the developer to specify the storage provider (PROVIDER_STORAGE_CLASS) and the node on which to schedule the application. (APP_NODE_SELECTOR)\n- The desired fio profile can also be specified. Currently, litmus supports simple [test-templates](https://github.com/ksatchit/litmus/tree/fio_test/tools/fio/templates), and is expected to grow to include multiple standard profiles. (FIO_TEST_PROFILE)\n- Certain simple test parameters such as the size of the test file (FIO_SAMPLE_SIZE) and duration of I/O (FIO_TESTRUN_PERIOD) can be specified as well, while the core I/O params continue to be housed in the templates.\n- The developer can choose to specify a comma-separated list of pods whose logs need to be collected for analysis of results, as well as the logs’ location on the host in the spec for the logger.\n\n```\nkarthik_s@cloudshell:~ (strong-eon-153112)$ cd litmus/tests/fio/\n\nkarthik_s@cloudshell:~/litmus/tests/fio (strong-eon-153112)$ cat run_litmus_test.yaml\n\n***\n\napiVersion: batch/v1\n\nkind: Job\n\nmetadata:\n\nname: litmus\n\nnamespace: litmus\n\nspec:\n\ntemplate:\n\n metadata:\n\n name: litmus\n\n spec:\n\n serviceAccountName: litmus\n\n restartPolicy: Never\n\n containers:\n\n - name: ansibletest\n\n image: openebs/ansible-runner\n\n env:\n\n - name: ANSIBLE_STDOUT_CALLBACK\n\n value: log_plays\n\n\n - name: PROVIDER_STORAGE_CLASS\n\n value: openebs-standard\n\n\n\n - name: APP_NODE_SELECTOR\n\n value: kubeminion-01\n\n\n - name: FIO_TEST_PROFILE\n\n value: standard-ssd\n\n\n - name: FIO_SAMPLE_SIZE\n\n value: \"128m\"\n\n\n - name: FIO_TESTRUN_PERIOD\n\n value: \"60\"\n\n\n command: [\"/bin/bash\"]\n\n args: [\"-c\", \"ansible-playbook ./fio/test.yaml -i /etc/ansible/hosts -v; exit 0\"]\n\n volumeMounts:\n\n - name: logs\n\n mountPath: /var/log/ansible\n\n tty: true\n\n - name: logger\n\n image: openebs/logger\n\n command: [\"/bin/bash\"]\n\n args: [\"-c\", \"./logger.sh -d 10 -r fio,openebs; exit 0\"]\n\n volumeMounts:\n\n - name: kubeconfig\n\n mountPath: /root/admin.conf\n\n subPath: admin.conf\n\n - name: logs\n\n mountPath: /mnt\n\n tty: true\n\n volumes:\n\n - name: kubeconfig\n\n configMap:\n\n name: kubeconfig\n\n - name: logs\n\n hostPath:\n\n path: /mnt\n\n type: Directory\n```\n\n## STEP 3: Run the Litmus fio test job.\n\nThe job creates the Litmus test pod, which contains both the test runner as well as the (stern-based) logger sidecar. The test runner then launches an fio test job that uses a persistent volume (PV) based on the specified storage class.\n\n```\nkarthik_s@cloudshell:~/litmus/tests/fio (strong-eon-153112)$ kubectl apply -f run_litmus_test.yaml\n\njob \"litmus\" created\n```\n\n## STEP 4: View the fio run results.\n\nThe results can be obtained from the log directory on the node in which the litmus pod is executed (By default, it is stored in _/mnt_). The fio & other specified pod logs are available in a tarfile (\\_Logstash**.tar\\_\\_).\n\n```\nroot@gke-oebs-staging-default-pool-7cc7e313-bf16:/mnt# ls\n\nLogstash_07_07_2018_04_10_AM.tar hosts systemd_logs\n```\n\nThe fio results are captured in JSON format with job-specific result sections. Below is a truncated snippet reproduced from the log for a sample basic rw run:\n\n```\n{\n \"jobname\": \"basic-readwrite\",\n \"groupid\": 0,\n \"error\": 0,\n \"eta\": 0,\n \"elapsed\": 61,\n \"read\": {\n \"io_bytes\": 28399748,\n \"bw\": 473321,\n \"iops\": 118330.31,\n \"runtime\": 60001,\n \"total_ios\": 7099937,\n \"short_ios\": 0,\n \"drop_ios\": 0,\n \"slat\": {\n \"min\": 0,\n \"max\": 0,\n \"mean\": 0,\n \"stddev\": 0\n },\n \"write\": {\n \"io_bytes\": 28400004,\n \"bw\": 473325,\n \"iops\": 118331.38,\n \"runtime\": 60001,\n \"total_ios\": 7100001,\n \"short_ios\": 0,\n \"drop_ios\": 0,\n \"slat\": {\n \"min\": 0,\n \"max\": 0,\n \"mean\": 0,\n \"stddev\": 0\n }\n }\n }\n}\n```\n\n## CONCLUSION\n\nHow is this different from doing an fio package installation on the Kubernetes nodes and running tests?\n\n- Running an fio Kubernetes job will offer better control to simulating actual application loads when used with resource limits.\n- The litmus fio jobs with various profiles can be included as part of a larger suite using the executor framework, thereby obtaining results for different profiles.\n- Litmus (as it continues to mature) will provide jobs that perform Chaos tests against storage while running different types of workloads. Running a fio job lends itself to that model.\n- Finally, it is a more “Kubernetes” way of doing things!\n\nLet us know your experience with using fio-based performance tests with Litmus. Any feedback is greatly appreciated!\n","notHasFeatureImage":true,"slug":"how-do-i-run-a-litmus-test-to-compare-storage-performance-on-kubernetes"},{"id":89,"title":"Kubernetes storage extensions to Weave Scope","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-6-2018","tags":["Open Source","Weave Scope","Node Disk Manager","Persistent Volume","Kubernetes"],"excerpt":"It was in Austin KubeCon 2017 that I first got a deep look at Weave Scope, and could not stop falling in love with it. The visualisation Scope provides into Kubernetes resources is simply amazing.","content":"\nIt was in Austin KubeCon 2017 that I first got a deep look at Weave Scope, and could not stop falling in love with it. The visualisation Scope provides into Kubernetes resources is simply amazing. It greatly simplifies the tasks of an Administrator in dealing with the clutter of Kubernetes components and helps directly go to the component of interest and start observing and managing it.\n\nBeing tasked with the goal of simplifying storage management for Kubernetes, my immediate thought was, why can’t we use Scope for Kubernetes storage? Of course, storage in Kubernetes is a developing area and new features are always coming but the existing adoption of Kubernetes persistent storage volumes(PVs) concept was already pretty large and we thought it warranted extensions to Scope to include PVs.\n\nSo we got to and with the help of [Alexis](https://twitter.com/monadic) and the Weave team — we started coding!\n\nWe set out multiple milestones for this journey:\n\n- The first one — get the persistent volumes (PVs), persistent volume claims (PVCs) and Storage Classes (SCs) into Scope\n- The second one — add snapshot/clone support and start monitoring the volume metrics\n- The third one — bring in the disk or SSD or similar as a fundamental resource that is being managed by the Administrator just like they might want to sometimes take a look at CPU and Memory\n\n## Persistent Volumes (PVs)\n\nMost of the time, Persistent Volume Claims (PVCs) are the entry points to increasing the storage. The number of PVCs will be about the same as the number of pods, or slightly less in a reasonably-loaded Kubernetes cluster. The administrator will benefit from having visibility of which POD is using which PVCs and the associated storage classes and PVs. This is especially true if they are using the storage capacity of the Kubernetes clusters themselves. Adding this visibility is precisely is what we did to start.\n\n![PVC-PV-SC-POD Relationship on Scope](/images/blog/pvc-pv-sc-pod.png)\n\nYou can see this new visibility in Scope by using the newly-created filter “Show storage/Hide storage” under the PODs section. This filter puts the storage components in perspective with the remaining pods and associated networked-data connections. Users can **Hide storage** when not interested, or to reduce clutter.\n\nWe received an enthusiastic welcome to the Scope community from the Weaveworks team. We also found encouragement from [Alexis](https://twitter.com/monadic) and plenty of technical help from [Bryan](https://twitter.com/bboreham) at Weaveworks. The first pull request (PR) was really about adding PV, PVC and Storage Class support, and was merged into the Weave Scope master recently ([https://github.com/weaveworks/scope/pull/3132](https://github.com/weaveworks/scope/pull/3132) ).\n\n![PV-PVC-SC Integration into Scope](https://blog.mayadata.io/hubfs/0_iYXgl-m8oxyXVs1s.gif)\n\n## Future work:\n\n### Snapshots and Clones\n\nCI/CD pipelines are the most active areas in which DevOps are finding stateful applications on Kubernetes to be immediately applicable. Storing the state of a database at the end of each pipeline stage, and restoring them when required, is a commonly performed task. The state of the stateful application is stored by taking snapshots of its persistent volumes and is restored by creating clones of persistent volumes. We believe that offering visibility and administrative capabilities to manage snapshots and clones in Scope is a natural next step.\n\n### Disk Management and Monitoring\n\nHyper-converged Infrastructure (HCI) has yet to find its rhythm with Kubernetes, largely due to a lack of fully-developed tools for disk management and monitoring. Kubernetes now has a well-accepted method to provision and manage volumes and attach them to disk management. Therefore, the enabling of HCI for Kubernetes will be improved by new tools such as [Node Disk Manager (NDM)](https://github.com/openebs/node-disk-manager), to which, incidentally (humble brag), we are also contributing. With Disk being the fundamental component for storage and the main participant in the chaos engineering of storage infrastructure, it helps to have it visualised and monitored in a proper way. In large Kubernetes clusters (100+) nodes, the disks will be in the thousands. Scope’s resource utilisation panel is a powerful tool that brings in the visibility of CPU and Memory utilisation at the Host, Container and Process level. This is a natural extension to add Disk Capacity, Disk performance (IOPS and throughput) to this resource utilisation tool. Our view is shown in the figure below, that Disk performance can be added.\n\n![Current View of the Resource Utilisation Tool on Scope](https://blog.mayadata.io/hubfs/0_9SozVWeQ2F69fDQO.gif)\n\nAnother important aspect of disk management is simply browsing from the application volume all the way to the disk where the data is stored. It is not possible to locate the actual disk of a persistent volume if the underlying storage is a cloud-disk such as EBS or GPD, but if it is a Kubernetes local PV or OpenEBS volume, the volume data vs. physical disks relationship can be identified. This will be useful while managing the hyper-converged infrastructure on Kubernetes.\n\n![(Future work) PODs/Disks and Nodes Relationship at Scope](https://blog.mayadata.io/hubfs/0_WJA8ii6NlaBoS94H.gif)\n\nThe above screens are a dirty implementation on a dev branch that is still in process. However, it provides a good, quick glimpse of how a POD’s volume is linked to the associated disks.\n\n_Weaveworks team recently started community meetings led by [Fons](https://twitter.com/2opremio), and it appears to be a great beginning of broader community involvement into the development of Scope. You can access the public meeting notes at_\n\n_[https://docs.google.com/document/d/103_60TuEkfkhz_h2krrPJH8QOx-vRnPpbcCZqrddE1s/edit?usp=sharing](https://docs.google.com/document/d/103_60TuEkfkhz_h2krrPJH8QOx-vRnPpbcCZqrddE1s/edit?usp=sharing)_\n\n## Summary:\n\nWeave Scope is a very useful tool for Kubernetes administrators for visualising and basic administration. With the addition of extensions being added, and a wider community being formed, Scope’s adoption will certainly increase and benefit the Kubernetes eco-system. We are looking forward to being an active contributor to this excellent visualisation tool.\n\nPlease provide any feedback here or in the next Scope community meeting. We will be there!\n\nThanks to [Akash Srivastava](https://medium.com/@srivastavaakash?source=post_page) and [Satyam Zode](https://medium.com/@satyamz?source=post_page).\n","notHasFeatureImage":false,"slug":"kubernetes-storage-extensions-to-weave-scope"},{"id":90,"title":"OpenEBS announces the availability of version 1.0","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-06-2018","tags":["CNCF","Kubernetes","OpenEBS","Storage"],"excerpt":"In this blog, I will touch upon what I have learned from users about Open Source CAS solutions like OpenEBS at KubeCon EU 19, followed by a quick note on the major changes in 1.0 and what is in the immediate roadmap.","content":"\nCongratulations and thanks to each one of you from the OpenEBS community for reaching this significant milestone!\n\nIn this blog, I will touch upon what I have learned from users about Open Source CAS solutions like OpenEBS at KubeCon EU 19, followed by a quick note on the major changes in 1.0 and what is in the immediate roadmap.\n\nBeing an active member of the OpenEBS Slack channel and from the usage statistics, I was cognizant of the vibrant user community. For instance, a recent usage report generated by MayaData since 0.8 was released, OpenEBS is running on — 340 different flavors of Operating Systems, 90 different flavors of Kubernetes and across 75 different countries.\n\n![OpenEBS Usage Report](https://cdn-images-1.medium.com/max/800/0*QKRCQN6eguXuHH6u)\n\nKubeCon EU 19, was by far the most interesting for me — to interact with some of the above users who are already running OpenEBS in production or in the process of evaluation. For those of you, curious about the kind of views expressed from users and the partner community about OpenEBS and Kubernetes Storage at large, here are some interesting ones:\n\n**Feedback from Users running OpenEBS in production.** The choice to adopt OpenEBS was made after evaluating and comparing other available solutions as can be seen from several stories shared by the users:\n\n![User feedback about adoption of OpenEBS](/images/blog/slack-snnipets.png)\n\nThis growing adoption is attributed to the following:\n\n- The major one being the architectural superiority of using Kubernetes to build highly portable distributed systems — and **not locking** all the applications into a single point of failure — by making them write into storage system run by some other group or provider. OpenEBS helps to build a storage solution where each application gets its own storage controllers. This is a perfect validation of the Container Attached Storage Category, which is explained in more detail at a [CNCF Blog](https://www.cncf.io/blog/2019/05/16/a-year-later-updating-container-attached-storage/).\n- Containerized distributed applications are much smarter than their legacy server-based counterparts and these new containerized applications don’t have the same dependency on the storage system as before. What applications need are **ala-carte** of Storage Solutions that OpenEBS offers as Storage Engines. Depending on the application needs, cluster administrators can decide what components of OpenEBS are really deployed. For example — does the cluster need to only have Local PVs or a full-fledged storage solution like cStor.\n- Of course the fact the OpenEBS itself is completely developed in user space and just runs on any platform easily. **No kernel taints required.** Yes, currently with 1.0 — there is a performance hit, but there are enough use cases where the current performance is good enough.\n\nWords just can’t express the joy of hearing directly from users on how OpenEBS has helped them. It is just amazing to see the diversity of users as well; from universities to financial corporations, startups to enterprises, fresher to seasoned SREs.\n\nPersonally, the most humbling moment for me at the KubeCon EU 2019, was when an end-user approached me just to thank for the efforts we have put into OpenEBS. He works at a University as a SysAdmin and he mentioned that using OpenEBS — he is able to spin up a self-contained Kubernetes Cluster with host storage that can be used by the stateful applications. This has helped in the number of support calls he receives from the University IT department when rolling out Stateful Applications that needed some hand-holding to provide the PVs.\n\nA similar sentiment is being expressed by large enterprises where the Infra team is supposed to roll out services to 100s of there application teams, in a more agile and uniform way than before. Some of the enterprises run Big Data pipelines at mammoth scale and have found OpenEBS to contains the right set of abstractions that can be extended and used.\n\n**Feedback from the growing community and partner ecosystem** that comprises of individual contributors, home users, enterprise solution architects, technology enthusiasts to hardware vendors included:\n\n- Solution Architects, DevOps, SREs >> I have put together a few scripts to make backups and restores with the Velero plugin for cStor volumes a little easier, since there are some parameters to remember and some additional steps required for both backups and restores other than using the basic Velero commands. I created a tool for [velero openebs](https://github.com/vitobotta/velero-openebs-backup) — with a few notes in a comment, in case anyone is interested\n- Storage Solution Engineers >> Hey, I know OpenEBS was not meant for this — but I think I made some changes can now serve Volumes from Kubernetes Cluster to Virtual Machines. A custom SDS, if you will, that is developed using OpenEBS.\n- Storage Vendors >> We are a storage devices company and OpenEBS can be used to run in an optimized way and help us shift storage boxes.\n- Kubernetes Managed Service Providers >> It will be cool to have OpenEBS available via the Operator Hub/Marketplace.\n\nBy the way, it is hard to gauge how much of the above discussions were really made possible by OpenEBS becoming part of the CNCF family. However, a major shift that I have observed since becoming a CNCF project, is that both end users and partners are more forthcoming to share their stories and integrate OpenEBS into their portfolio. _For anyone considering donating their projects to CNCF, I highly recommend it **—** there is a steep learning curve that helps with Governance of the project, by just being part of the efforts to propose the project into CNCF._\n\nClearly, these days the discussions at KubeCon EU are not around What or Why OpenEBS? but around When a specific feature is going to be made available. While the top requests with regards to missing features were around CSI support, automation of the operations like — disk provisioning and capacity expansion and performance improvements — they were really not blocking users from running OpenEBS in production. There are lots of users out there, that have found the current performance offered by the OpenEBS Data Engines was good enough for their use cases, and they have gone ahead and implemented playbooks and tools for performing maintenance and monitoring operations.\n\nPost KubeCon, having known and also met in person some of these early adopters that are running OpenEBS in production with a set of home-grown solutions, we felt a sense of responsibility to release 1.0 at the earliest and promise to provide long term support on the current feature set. As we progress with implementing additional functionality around operators, CSI, supporting the latest from Kubernetes Storage, and so forth we are also making a commitment that the current version will be supported.\n\nHence, for 1.0 — we changed our gears a bit to focus on helping address the feedback received on the existing feature sets for all the three data engines now supported by OpenEBS and deployed in production- Jiva, cStor Data Engines and Local PV.\n\nSome significant changes in 1.0 are as follows:\n\n**Enhanced Lifecycle Management of Block Devices**. The component of OpenEBS — NDM — Node Disk Manager or Node (Storage) Device Manager as we call it now, has been enhanced to support Block Device Claim (BDC). A new NDM Operator has been introduced in 1.0, that helps to request and reserve a Block Device before using that for either creating Local PV or cStor Pool. NDM is also being used independently of OpenEBS Storage Engines, and the ability to claim a Block Device — similar to PVC — enables the sharing of the block devices without stepping on each other.\n\nBlock Device(BD) is a new CR — which is a preferred way to represent a storage device than a Disk CR ( which was used in earlier releases.) Disk CR are also present — to allow for backward compatibility. The cStor Data engines and Local PVs that typically use block devices have been enhanced to use BDC/BD in place of Disk.\n\nWhen a BDC is deleted, the BD is released. And before it can be claimed by another BDC, the NDM takes care of deleting the data that has been written from the previous consumer. The cleaner utility used is derived from the Kubernetes Local PV cleaner jobs.\n\n**Enhance the OpenEBS Local PV to be used with Block Devices.** In the previous release, we introduced Local PVs that can be used with a hostpath, in this release, we have added support to create a Local PV directly on a Block Device (discovered by NDM). The Block Device can either be a raw block device (Example: GPD) — in which case it will be formatted (Example: Local SSDs on GKE) with the requested filesystem type or already mounted block device.\n\nAlso included in this release are:\n\n- Bug fixes reported around the cStor and Jiva data engines that surface after running OpenEBS on nodes that experience frequent reboots.\n- Additional debugging tools for cStor and Jiva engines, jivactl being the significant one that will help with clearing up of older snapshots. This tool has to be used to clear older snapshots to avoid hitting a cap limit of 255 snapshots in older releases.\n- Added few more Prometheus monitoring metrics\n- Added several e2e tests and included more applications into the OpenEBS CI pipeline maintained by [MayaData](https://mayadata.io/) at [openebs.ci](https://openebs.ci/)\n\nFor detailed changes summary, please check the [OpenEBS 1.0 Changelog](https://github.com/openebs/openebs/wiki/Change-Summary----v1.0).\n\nPlease refer to the [OpenEBS Documentation](https://openebs.io/docs?__hstc=216392137.ed92f0691adfb1cbf08ea329504224a3.1580116629364.1580116629364.1580116629364.1&__hssc=216392137.1.1580116629364&__hsfp=3765904294) to get started.\n\nOpenEBS 1.0 — ships with multiple Data Engines to choose from depending on your application needs:\n\n- OpenEBS Jiva PVs — can be used if your kubernetes nodes don’t have the capability to add raw block devices, but have extra capacity available on the host filesystem. This is the first and the longest running in production with lots of Kubernetes tunables available like — to customize the location where data is saved, the specific nodes on which data needs to be replicated — within or across availability zones, setup the volumes for thin provisioning, and so forth.\n- OpenEBS cStor PVs — is the preferred option when your nodes have raw block devices. cStor Data Engine continues to be a preferred solution for use cases that require instant snapshot and clone of volumes.\n- OpenEBS Local PVs — best suited for applications that can do their own replication and require high performance. OpenEBS Local PVs can work with either hostpath or with block devices that are already attached to the nodes.\n\nThere are a lot of options available to customize from the type of block devices that can be used for OpenEBS volumes to customizing the resources allocated to the Storage Pods.\n\nOpenEBS has turned a new leaf in its journey by releasing version 1.0. Thank you for all the support and love that you have shown along this journey. We are forever more committed to learn from you and help you realize the Data Agility that Cloud Native promises to offer.\n\nOur immediate focus is on getting OpenEBS ready for CSI and enhancing the OpenEBS Operators — for managing the Day 2 operations like dealing with Kubernetes upgrades, ASG, expansion of cStor Pool capacity or migrating them to new nodes and so forth. Development on these items is being tracked in the Kubernetes style at [Product Planning Sheet](https://docs.google.com/spreadsheets/d/1bbphUqbxShBhgr1VHaEQUzIGMaJJacPNKc1ckNXU1QE/edit).\n\nThanks to CNCF, we have the following mailing lists to connect apart from the [Slack](https://openebs.org/community).\n\n- For OpenEBS project updates — subscribe to [https://lists.cncf.io/g/cncf-openebs-announcements](https://lists.cncf.io/g/cncf-openebs-announcements)\n- For interacting with other OpenEBS users, you can either join the [Slack](https://openebs.org/community), raise issues on [Github](https://github.com/openebs/openebs/issues) or send an email to [https://lists.cncf.io/g/cncf-openebs-users](https://lists.cncf.io/g/cncf-openebs-users)\n\nAs always, I am eager to learn from you. Please hit me up on [twitter (@kiranmova)](https://twitter.com/kiranmova), [slack](https://openebs.org/community) or on here in the comments.\n\nThanks to [Murat Karslioglu](https://medium.com/@muratkarslioglu?source=post_page). [Public domain](https://creativecommons.org/publicdomain/mark/1.0/).\n","notHasFeatureImage":true,"slug":"openebs-announces-the-availability-of-version-1.0"},{"id":91,"title":"Git freedom on Kubernetes","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"05-06-2018","tags":["DevOps","Git","Gitlab","Kubernetes","OpenEBS"],"excerpt":"Here is one of the fastest ways to get your private repository with Gitlab up and running on your Kubernetes environment — Let’s “Make DevOps lifecycle private again”","content":"\n#### Steps to easily run GitLab\n\nAfter Microsoft announced the acquisition of GitHub, many developers raised concerns on social media about Microsoft’s history of unsuccessfully running the acquired businesses like Skype, Nokia’s handset business, Navision and other 150 companies (you probably haven’t noticed) they have swallowed up over the years.\n\nOther than keeping the developer’s life-support plugged, one of the biggest concerns is that MS would use its power on GitHub repositories to analyze trends among software development in order to launch competing products. Fears that GitHub privacy may be in jeopardy have already led many developers to jump off the ship or consider alternatives. GitLab’s publicly available [status graphs](https://t.co/svpWpI0Rb2) show spikes of 70x increase in imported repositories (average 100 vs 7.5K), a validation of increased user apprehension.\n\nHere is one of the fastest way to get your private repository with Gitlab up and running on your Kubernetes environment — Let’s “**Make DevOps lifecycle private again**” ©\n\nCurrently, the easiest and recommended way to install GitLab on Kubernetes is using the [Gitlab-Omnibus](https://docs.gitlab.com/ee/install/kubernetes/gitlab_omnibus.html) Helm charts.\n\nGitlab-Omnibus deploys every feature a small deployment would require including the [Container Registry](https://docs.gitlab.com/ee/user/project/coThis year I have attended a number of tech events and in terms of size, organization, and especially the content — Next ’18 is so far my favorite.ntainer_registry.html#gitlab-container-registry), [load balancer (NGINX)](https://github.com/kubernetes/ingress/tree/master/controllers/nginx), [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/), and [Runner](https://docs.gitlab.com/runner/).\n\n#### **Prerequisites**\n\nMinimum requirements for a multi-node cluster:\n\n**Hardware**\n\n- **Boot node:** 1x 1+ core(s) >= 2.4 GHz CPU, 4GB RAM, >=100 GB disk space\n- **Master node:** 1 or 3x 2+ cores >= 2.4 GHz CPU, 4+GB RAM, >=151 GB disk space\n- **Worker node:** 3x 2+ cores >= 2.4 GHz CPU, 4+GB RAM, >=100 GB disk space\n\nSince I’m not planning to run anything heavy, I’ll be using 3 nodes, and will install Master, Proxy, and Workers an all 3.\n\n**Software**\n\n- [Ubuntu 16.04 LTS](https://www.ubuntu.com/download/server) (RHEL 7.x is also supported)\n- Docker 1.12 to 17.03\n- Kubernetes 1.7+ Cluster (You can use [IBM Cloud Private 2.1.0.2](http://containerized.me/how-to-install-openebs-on-ibm-cloud-private/) or [Red Hat OpenShift Origin](http://containerized.me/how-to-install-openebs-on-openshift/))\n- [kubectl](https://github.com/kubernetes/kubectl)\n- Helm client\n- A [GitLab Omnibus](https://docs.gitlab.com/omnibus/) Pod, including Mattermost, Container Registry, and Prometheus\n- An auto-scaling [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor\n- [Redis](https://github.com/kubernetes/charts/tree/master/stable/redis)\n- [PostgreSQL](https://github.com/kubernetes/charts/tree/master/stable/postgresql)\n- [NGINX Ingress](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress)\n- [OpenEBS](https://github.com/openebs/openebs) persistent volumes for Data, Registry, Postgres, and Redis\n\nThe Kubernetes instructions described below using Helm are generic and should work on all other platforms.\n\n**Installing GitLab and OpenEBS using the Helm Chart**\n\nGitLab depends on stateful applications like Redis and PostgeSQL, and requires persistent volumes for its data and the registry. Here, I will simplify the storage provisioning using OpenEBS.\n\nFirst, install OpenEBS using the chart.\n\n```\nhelm install — name ‘openebs-gitlab-test’ stable/openebs\n```\n\nOptional: If you would like to customize your OpenEBS installation you can also use a copy of the [value.yaml](https://raw.githubusercontent.com/kubernetes/charts/master/stable/openebs/values.yaml) file from the OpenEBS chart and modify parameters listed [here](https://github.com/kubernetes/charts/tree/master/stable/openebs).\n\n```\nhelm install — name ‘openebs-gitlab-test’ -f values.yaml stable/openebs\n```\n\nNext, add the predefined storage classes.\n\n```\nkubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml\n```\n\nThere are many ways to enable OpenEBS for use by GitLab. The fastest is by making one of the OpenEBS storage classes a default StorageClass:\n\nList available OpenEBS storage classes in your cluster.\n\n```\nmurat@icpnode1:~$ kubectl get sc\nNAME PROVISIONER AGE\nopenebs-cassandra openebs.io/provisioner-iscsi 18d\nopenebs-es-data-sc openebs.io/provisioner-iscsi 18d\nopenebs-jupyter openebs.io/provisioner-iscsi 18d\nopenebs-kafka openebs.io/provisioner-iscsi 18d\nopenebs-mongodb openebs.io/provisioner-iscsi 18d\nopenebs-percona openebs.io/provisioner-iscsi 18d\nopenebs-redis openebs.io/provisioner-iscsi 18d\nopenebs-standalone openebs.io/provisioner-iscsi 18d\nopenebs-standard openebs.io/provisioner-iscsi 18d\nopenebs-zk openebs.io/provisioner-iscsi 18d\n```\n\nEither create your StorageClass or pick one of the predefined classes. _openebs-standard_ creates 3 replicas and is an ideal candidate here to be used for most of the stateful workloads. Let’s mark this StorageClass as default.\n\n```\nkubectl patch storageclass openebs-standard -p ‘{“metadata”: {“annotations”:{“storageclass.kubernetes.io/is-default-class”:”true”}}}’\n```\n\nNo verify that your chosen StorageClass is indeed the **default**.\n\n```\nmurat@icpnode1:~$ kubectl get sc\nNAME PROVISIONER AGE\nopenebs-cassandra openebs.io/provisioner-iscsi 18d\nopenebs-es-data-sc openebs.io/provisioner-iscsi 18d\nopenebs-jupyter openebs.io/provisioner-iscsi 18d\nopenebs-kafka openebs.io/provisioner-iscsi 18d\nopenebs-mongodb openebs.io/provisioner-iscsi 18d\nopenebs-percona openebs.io/provisioner-iscsi 18d\nopenebs-redis openebs.io/provisioner-iscsi 18d\nopenebs-standalone openebs.io/provisioner-iscsi 18d\nopenebs-standard (default) openebs.io/provisioner-iscsi 18d\nopenebs-zk openebs.io/provisioner-iscsi 18d\n```\n\nNext, we can install the GitLab-ce chart. It is recommended to save your configuration options in a values.yaml file for future use.\n\n```\nwget https://raw.githubusercontent.com/kubernetes/charts/master/stable/gitlab-ce/values.yaml\n```\n\nEdit the _values.yaml_ file and at minimum, add the **externalUrl** field. Otherwise, you’ll end up with a non-functioning release.\n\nHere is how my _values.yaml_ file looks like after these changes:\n\n```\nimage: gitlab/gitlab-ce:9.4.1-ce.0\nexternalUrl: http://containerized.me/\nserviceType: LoadBalancer\ningress:\nannotations:\nenabled: false\ntls:\nurl: gitlab.cluster.local\nsshPort: 22\nhttpPort: 80\nhttpsPort: 443\nlivenessPort: http\nreadinessPort: http\nresources:\nrequests:\nmemory: 1Gi\ncpu: 500m\nlimits:\nmemory: 2Gi\ncpu: 1\npersistence:\ngitlabEtc:\nenabled: true\nsize: 1Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\ngitlabData:\nenabled: true\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\npostgresql:\nimageTag: “9.6”\ncpu: 1000m\nmemory: 1Gi\npostgresUser: gitlab\npostgresPassword: gitlab\npostgresDatabase: gitlab\npersistence:\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\nredis:\nredisPassword: “gitlab”\nresources:\nrequests:\nmemory: 1Gi\npersistence:\nsize: 10Gi\nstorageClass: openebs-standard\naccessMode: ReadWriteOnce\n```\n\nNow, install the chart.\n\n```\nhelm install — name gitlab-test -f values.yaml stable/gitlab-ce\n```\n\nList the pods and confirm that all pods are ready and running.\n\n```\n$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\ngitlab-test-gitlab-ce-dd69cdf4b-69vmb 1/1 Running 0 11m\ngitlab-test-postgresql-75bf9b667d-lwj2b 1/1 Running 0 11m\ngitlab-test-redis-998998b59-hzztj 1/1 Running 0 11m\nopenebs-gitlab-test-apiserver-68fc4488fd-jf8gz 1/1 Running 0 1h\nopenebs-gitlab-test-provisioner-7dfdf646d8–9wpmg 1/1 Running 0 1h\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-ctrl-74d4b59c9f-bjtg2 2/2 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-6ds26 1/1 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-99mbh 1/1 Running 0 11m\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800-rep-64f56667d-d8d4z 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-ctrl-bd7cff65f-ph8dr 2/2 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997–2lm4x 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997-jldjs 1/1 Running 0 11m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800-rep-595dd9c997-kzlrc 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-ctrl-668f5988c5-hv8vb 2/2 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-hsn49 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-lj64g 1/1 Running 0 11m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800-rep-74974f6644-z6kfd 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-ctrl-585cf7c97d-58pnq 2/2 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-5bzn6 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-9dz5f 1/1 Running 0 11m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800-rep-79d658d94c-snkfb 1/1 Running 0 11m\n```\n\nGet the list of persistent volumes.\n\n```\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-cb0fc1b2–6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-postgresql openebs-standard 17m\npvc-cb1064ee-6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-redis openebs-standard 17m\npvc-cb111261–6904–11e8–9f57–06a0a9acf800 10Gi RWO Delete Bound default/gitlab-test-gitlab-ce-data openebs-standard 17m\npvc-cb11a791–6904–11e8–9f57–06a0a9acf800 1Gi RWO Delete Bound default/gitlab-test-gitlab-ce-etc openebs-standard 17m\n```\n\nYou can see above that four persistent volumes were created (**postgresql, redis, gitlab-ce-etc, gitlab-ce-data**), and each volume is protected by 3 replicas.\n\nNow go to the external endpoint address you have defined and start using GitLab after you set your new password.\n\n![](https://lh4.googleusercontent.com/9UnAe3ZuKt_weq1IbxOrgA_JQMpXE2ZCd80PgDxIodeUdFslr-wCt2DUjbWyoERYWa6RKht8JYvihV-_dQS0EYArc4dJhkPPtN0cGPNfYcDsiHgtjS7unCLOW9MTDre79AjZ660xm6IN94OPew)\n\nNow click on **Create a project**, then import your existing project from GitHub and start using GitLab.\n\n![](https://lh4.googleusercontent.com/CDe7SDXnBmCL5IGVTIOYATjzjZN2iMPsZaVBmuY3-l6qXFX8xReeU6M234eX0ELY1Pips6JfR1FJb4rzfL_d53KLDon0MrzBKqQvuBslQDboCw1yPehiKrSf771PMy79ckmPdLGWnhmijDFkVg)\n\n![](https://lh6.googleusercontent.com/AFqy2l5MohpC1kCk5k2yFoZA90qJGabfUymqmMmI0kFqcpgqXnrapoYCs1dMfrFDqKj-37ncNvoCe7Kf8UfCQq6VRvmFMK742abC58ju6TiRSUk2yeq1OtMBZWPMd3pqyQWawBDgUcSpSZ8Djg)\n\n---\n\nHopefully, this helps anyone who is motivated to reexamine their approach to Git to quickly and easily start running GitLab on Kubernetes. Thank you for reading, and please provide any feedback below or via Twitter — [**@**muratkarslioglu](https://twitter.com/muratkarslioglu)_Originally published at _[_Containerized Me_](http://containerized.me/git-freedom-on-kubernetes/)_._\n","notHasFeatureImage":false,"slug":"git-freedom-on-kubernetes"},{"id":92,"title":"Berlin K8s meetup retrospect","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"31-05-2018","tags":["OpenEBS","Kubernetes","DevOps"],"excerpt":"Last week, I was invited to give a talk about OpenEBS during the Kubernetes meetup in Berlin. The event was hosted by the friendly folks at Profitbricks, who once again I want to thank for the lovely venue and experience.","content":"\nLast week, I was invited to give a talk about OpenEBS during the Kubernetes meetup in Berlin. The event was hosted by the friendly folks at Profitbricks, who once again I want to thank for the lovely venue and experience. Matt Baldwin, from our friends at StackPoint Cloud, was once again an organizer — thank you Matt for your ongoing support of the community (and of OpenEBS as well).\n\nAs we went over some of the nitty-gritty details on how we are building OpenEBS, I received a few questions that I thought deserved an extended response. I will address one of those here.\n\nA common question was: “If I’m running on an __ system, what is the benefit of OpenEBS?”\n\n**_So, let’s dive into one of the biggest questions— what if you are running on AWS and you are using EBS volumes, why use OpenEBS on top?_**\n\nFirst of all, we already made the implicit assumption that if you were to run an application managed by Kubernetes, you are using EBS volumes. But, this assumption does not have to be true. In fact, if you want fast and performant storage, AWS suggests that you use optimized storage instances, meaning _not using EBS_ volumes at all.\n\nThe EBS volumes are not the fastest thing you can get from AWS, but they do provide replication across zones and snapshots. This brings me to the second part, a snapshot, in AWS, is stored in S3. That bill (depending on dataset size) can quickly and easily pile up with the number of snaps. With OpenEBS, we store the snapshot locally on the volume itself, and we don’t require you to “copy” this into S3 or a similar location. Also, consider the amount of time it takes for a snapshot to be created in AWS; it can take hours and the snapshot restores require you to attach a new EBS volume. Moreover, if you do these types of things, you have to “fiddle” with the OS to get it attached, something that OpenEBS handles for you, making snapshots instantly and allowing you to rollback immediately.\n\nThere is also the matter of granularity; with OpenEBS, you can control everything per application or even per workload. You can fine-tune the replication factors (or not have it all) and you can use just one EBS volume for all your apps since OpenEBS handles the slicing and dicing for you.\n\nSo, this was about using EBS volumes, I hope the advantages are apparent and easy to understand. This brings data agility to your applications, not just in AWS, but also across clouds: public, private or hybrid. The developer does not have to make any changes to his or her YAML, and can write once and run everywhere, precisely what containers are supposed to do. Do not let others take that freedom away from you =)\n\nBack to the non-EBS volumes, what AWS refers to as “instance store.” More information from Amazon:\n\n![Instance stores](/images/blog/instance-stores.png)\n\nAs the docs point out, when you use this, you get high speed with zero features. With OpenEBS, you get high speed and all of the features if you delegate control of the “instance store” to OpenEBS. An interesting note, the instance store comes with the price of the machine instance itself, so there are no extra charges on your bill.\n\nIn a future blog I’ll cover some other common questions that arose — and I’m also going to continue talking about the shift of IO and so forth to the user space, and what it may mean for Container Attached Storage and running IO intensive workloads in general.\n\nFeel free to ask any questions below, or maybe I’ll see you on the [OpenEBS slack community](https://slack.openebs.io/?__hstc=216392137.c2ca19ae0ec72f00a9ae3bf6f8a512a3.1579861713935.1579861713935.1579861713935.1&__hssc=216392137.1.1579861713935&__hsfp=3765904294) where we are discussing many related subjects.\n","notHasFeatureImage":false,"slug":"berlin-k8s-meetup-retrospect"},{"id":93,"title":"Container Attached Storage (CAS) — Taking off nicely","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-05-2018","tags":["CAS","Kubernetes","Stateful Applications","OpenEBS","MayaOnline"],"excerpt":"I had the fortune of presenting to a group of brilliant folks at SNIA India SDC event last week. This event being in Bangalore, I could sense the heat emanating from technology savvy brains mostly from the storage companies like DELL-EMC, NetApp, RedHat storage, HP storage etc","content":"\nI had the fortune of [presenting](https://www.slideshare.net/OpenEBS/openebs-cas-sdc-india-2018) to a group of brilliant folks at SNIA India SDC event last week. This event being in Bangalore, I could sense the heat emanating from technology savvy brains mostly from the storage companies like DELL-EMC, NetApp, RedHat storage, HP storage etc and the questions at the end of the presentation were a proof of it. Lots of great questions on the lines of “Nice architecture, I never knew storage can be run completely in user space”, “CAS — great topic, I understand the value and benefits, but are not you adding too many containers to the equation?”. In the interest of answering those questions in detail, I thought let me take the most commonly asked ones and answer them in this short article.\n\n## Just a quick recap on CAS:\n\n[Container Attached Storage (CAS)](/docs/concepts/cas) is a new storage architecture to run the entire storage software in containers and hence in user space. This architecture has many benefits, primary one being “a dedicated storage controller per application” and bring in the possibility of hardening the storage controller for a given application workload. Read more on the benefits at the [CNCF blog](https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/). A typical CAS architecture example is shown below.\n\n![CAS architecture with controller and replica pods for each application](https://cdn-images-1.medium.com/max/800/1*4dJDmPbxxrP-fZK7NZZmYg.png)\n\nNow back to the CAS FAQ, here are a couple:\n\n### Q1: Well, how can you run entire storage in user space? I thought it can never be run in user level because of performance reasons.\n\nFor more technical answer, read our [CTO’s blog](/blog/the-mule-and-the-flash-going-for-a-run) on this precise topic 🙂\n\nShort answer is, we think storage should take advantage of container technology, so in CAS, we run storage as a microservice. The performance will be taken care by using technologies like SPDK, VPP and also taking advantage of the abundant availability of compute cores on the node.\n\nReality is that, linux kernel, as it is today, cannot deliver all the IOPS from the underlying NVMe flash disks. Do we think kernel can deliver 12 Million IOPS from the 24 NVMe disks ? Of course not, not today !!\n\nIn summary, higher performance can only be delivered using CAS architecture, SPDK, VPP and using more CPU cores.\n\n![CAS with SPDK leads to higher performance](https://cdn-images-1.medium.com/max/800/1*aKjepAaB5sIZF-hOq_dxIg.png)\n\n### Q2: In CAS, you are adding more containers to the cluster because of storage. Isn’t that increasing the compute needs of the Kubernetes cluster?\n\nCAS enables native hyper convergence capability on Kubernetes. You don’t need an external storage array to manage the storage/data needs of applications on K8S, thats a lot of saving of CPU and hardware. With CAS, the overall TCO reduces as the additional storage (local disks), CPU and software (CAS) are provisioned on the same K8S cluster nodes and avoids the need of expensive external storage arrays.\n\nAs for the the question of dealing with large number of containers is concerned, there are nice tools like Weave Scope which you can use on the local K8S.\n\n![Application relationship with storage (PVC, PV, SC, and CAS)](https://cdn-images-1.medium.com/max/800/1*RQYjI0MdsXf1kj8AGqLJZA.png)\n\n## Conclusion:\n\nWe continue to say — With CAS, storage fades away as a concern !! Join our [slack community](https://slack.openebs.io) to find more !!\n","notHasFeatureImage":false,"slug":"container-attached-storage-(cas)-—-taking-off-nicely"},{"id":94,"title":"My First Contribution to OpenEBS \\#OSS","author":"Vipul Gupta","author_info":"Enter a short bio? No, thank you. If you like to find me or my content, then I would be at http://mixstersite.wordpress.com/","date":"28-05-2018","tags":["OpenEBS","Go","Golang","Summer Hackfest","Open Source"],"excerpt":"Writing documentation for any project is tough, be it big or small, propriety or open-source. Rewriting and improving it is even tougher. Let no one tell you any different.","content":"\nWriting documentation for any project is tough, be it big or small, propriety or open-source. Rewriting and improving it is even tougher. Let no one tell you any different. Take it from someone who scoured through 6000+ lines of documentation written in Markdown spread across 20 or so files and been able to fix about 900 odd lines of them for a major open-source project. That’s how my story goes if you want to know about my first contribution to OpenEBS.\n\nHonestly, I have little experience in Go, I have never been inclined to use it much. As I am already comfortable, writing code in Python for about a year now. But that doesn’t stop me from learning new languages and technologies. Hence my utmost dedication to cure myself of the “Too Much too Learn” Syndrome. I went to the **Women Who Go meetup** in New Delhi, India for a [Go 101 workshop](https://twitter.com/vipulgupta2048/status/977893034808434689?s=09). And that’s how I got to know about the open-source project, OpenEBS, and their exciting Summer HackFest.\n\n[OpenEBS](https://openebs.io/) is an open source storage platform that provides persistent and containerized block storage for DevOps and container environments. OpenEBS is also tightly integrated with Kubernetes and is a part of Kubernetes-Incubator project. And works with all popular workloads such as PostgreSQL, Redis, Jenkins etc.\n\nNow, OpenEBS is still under active development from the team. Hence comes in, [Summer HackFest](https://openebs.io/hackfest). **Summer HackFest** is a unique opportunity for new contributors and people working with Go to develop and fix issues for OpenEBS. By contributing to the Hackfest, you get community access, useful knowledge on working with new technologies, learn new skills and along the way win cool goodies, swag. Also, the chance to take home the grand prize, a laptop. HackFest starts out in April 2018. Being open-source, all contribution was needed to be submitted to [GitHub](https://github.com/openebs/openebs). I did and oh man, I had a good time writing code for OpenEBS. Sounds good. Want to contribute?\n\n---\n\n## Let’s get you started\n\nRefer to this [quick guide](https://github.com/search?utf8=%E2%9C%93&q=org%3Aopenebs+is%3Aissue+label%3Asummerhack+label%3Akind%2Funit-test&type=) to get started. You would also need to be familiar with the basics of Go, hence you can refer to the widely recommended tutorial [https://tour.golang.org/](https://tour.golang.org/), for the same. After that, make sure to join the OpenEBS Slack channel to resolve all doubts and questions that you might have. The link can be found here in the [contributing document](https://github.com/openebs/openebs/blob/master/CONTRIBUTING.md) on their repository. Read the document (I re-wrote it) And you are committed.\n\nGo to their website that I linked above, and figure out which section of issues you would like to contribute. Once done. Head over there, and do your thing. Push the changes, make that PR and rest is history.\n\n## Well, sure thing but how was your experience?\n\n![Contributors to OpenEBS Hackfest](/images/blog/hackfest-contributors.png)\n\nMy experience has been great so far while contributing to OpenEBS (_I am still actively contributing to it_). Everything was a breeze. They have very flexible rules and easy guidelines, no hard checks with the CI. The reviewers are very helpful, and would assist you in every possible manner ( Shoutout to Kiran Mova !!) I have been working on solving issues in the documentation of OpenEBS and vagrant files that are being used in the project.\n\nEarlier, I also attended a webinar OpenEBS conducted on Introduction to contributing and solving issues of Maya and scope, which was very informative and helpful. Hence goes on to show their dedication to helping their contributors even further then most people would go. The codebase could be quite intimidating at first, but with time and the helpful answers provided by the very vibrant and active community will keep you going. I can tell you for a fact, that the journey is the reward here.\n\nI asked so many questions, and never got a blowback. I really appreciated all the support shown to me by the community. The environment is beginner-friendly and very welcoming, great for all new contributors. It couldn’t get any better with the incentives they add to it for contributing.\n\nI honestly had a lot of fun while contributing. One of the most notable [pull requests](https://github.com/openebs/openebs/pull/1511) of mine to Summer Hackfest was about rewriting and improving the documentation, which received a whopping 392 comments. They were sweet enough to put my name on the website with the fellow contributors and made me the star contributor of the week. Woohoo !! They also awarded me with a book as a token of their appreciation. Thanks, OpenEBS !!\n\nAnd that’s it for this time, I guess. For all my PR’s to Summer HackFest, [click here](https://github.com/search?utf8=%E2%9C%93&q=org%3Aopenebs+is%3Apr++label%3Asummerhack++author%3Avipulgupta2048+is%3Amerged&type=Issues). I will update the blog post with a picture of the goodies once I receive them (_So excited_). I highly recommend heading over to [https://openebs.io/hackfest](https://openebs.io/hackfest), to take part in this golden opportunity to contribute to such a brilliant project and being a part of their community.\n\n---\n\nVipul Gupta ([vipulgupta2048](http://www.letmegooglethat.com/?q=vipulgupta2048)) posts occasional on his blog, [Mixster](http://www.mixstersite.wordpress.com/). Check him out there for more updates like these.\n","notHasFeatureImage":false,"slug":"my-first-contribution-to-openebs-\\#oss"},{"id":95,"title":"Using Kubernetes Custom Resources for Microservices IPC","author":"Ganesh Kumar","author_info":"Gopher, Open Source Contributor, Thinker, Health enthusiast","date":"11-05-2018","tags":["Kubernetes","Golang","OpenEBS","Microservices"],"excerpt":"This blog talks about why I used Custom Resources as a way for communication between different microservices (aka Kubernetes Pods).","content":"\nThis blog talks about why I used [Custom Resources](https://kubernetes.io/docs/concepts/api-extension/custom-resources) as a way for communication between different microservices (aka Kubernetes Pods).\n\nOpenEBS is a fully containerized storage solution running within Kubernetes. Infact [OpenEBS](https://openebs.io/docs) extends the Kubernetes cluster functionality to manage storage and stateful workloads.\n\nOpenEBS has an operator (or orchestration component) called [Maya](https://github.com/openebs/maya) (magic) that relays Volume management operations to several storage engines. OpenEBS already supports [Jiva](https://github.com/openebs/jiva) based storage engine. The purpose is to plugin a more independent [cStor](https://github.com/openebs/cstor)based storage engine (making use of zfs on userspace).\n\nI essentially have a user interacting with OpenEBS Maya using PV, PVC, StorageClasses etc. and OpenEBS Maya interacts with cStor Pods. I will be focusing on the design considerations for interactions between OpenEBS Maya and cStor Pods.\n\n![Architecture](/images/blog/architecture.png)\n\n_1 — Shows user creating StoragePoolClaim (SPC) CR, with details like the number and type of pools to be created. Let us consider, SPC specifies a cStor pool should be created._\n\n_2 — Maya has a custom controller that watches for the SPC CR and it will go ahead and create the cStor Pods with a cstor-pool container and a side-car cstor-pool-mgmt that has a CLI interface for creating pools and volumes. Side-car container (following the Kubernetes Ambassador Pattern), helps in translating the Pool and Volume operations triggered by Maya into the corresponding CLI commands._\n\n_3 — Depending on the user’s request for creating a pool or a volume, Maya will create CStorPool and CStorVolumeReplica CRs respectively. Note that, I could have had the cstor-pool-mgmt container expose an API service that Maya could have invoked. Instead, I decided to use CRs and I will explain why in the following sections._\n\n_4 — cstor-pool-mgmt sidecar application watches for CRs of CStorPool and CStorVolumeReplica and performs pool — volume operations._\n\n**One of the core design constraint while deciding on inter-pod communication between Maya and CStor Pod is that:** _When the user requests for a Volume, the cluster state may not be fully ready to satisfy all the criteria — for example, User requests for 3 replicas but there are only 2 nodes running. The request should be cached and a third replica has to be provisioned whenever a new node gets added to the cluster._\n\nDevelopers normally go with an approach to make a REST/gRPC call to the receiver and store in a database, running in separate pod/statefulset. But OpenEBS thinks beyond that.\n\nNow consider, that I had used the traditional approach of using a REST/gRPC method of interactions between Maya and CStor Pods, then Maya would have to implement/consider cases like:\n\n- Where to store the state of current request, as the request can’t be serviced immediately depending on the cluster state. This is required to handle the case where the node running Maya can itself go down.\n- When working in a scaled environment, when there are multiple Maya pods, who gets to service the requests and when one of the pod goes down, should the other take it over or not?\n- How to handle the case where, Maya sends the request to the CStor container and then it goes boom (after all this is Kubernetes Cluster and they are supposed to handle all kinds of Chaos), who handles the results of the operation at CStor. In other words, how to implement a 2-phase commit?\n\n_However, if you look at my design constraint, doesn’t it sound similar to how a Kubernetes deployment with 3 replicas work?_ The user defines a desired state (in this case Maya) and the controllers make it happen eventually. So, why not just, be a roman when in rome.\n\n### Thats exactly what OpenEBS does!\n\nOpenEBS goes with `watch`-er approach. i.e., watch [**k8s custom resource definition**](https://kubernetes.io/docs/concepts/api-extension/custom-resources). If a pool (virtual disk) must be created on top of the actual disk, Maya creates a custom resource (named CStorPool), and a pod running cstor-pool-mgmt watcher gets an event for corresponding resource request and starts performing pool related operations. Cool, isn’t it?\n\n### Where does the custom resource get stored?\n\nkubernetes etcd. You pronounced it right, that’s `yetsed`, :-)\n\n### How is custom resource efficient?\n\n- Storing critical details in a separate pod-database, leads to pod level consistency. Storing in etcd leads to **cluster level consistency**.\n- Even if the **receiver is not running**, when the request is generated, the receiver-watcher gets an event, as and when it starts running.\n- Users can access the resources via **k8s cli** — `kubectl get `\n- Update the status of request on the same custom resource.\n\n### What is the problem with custom resource?\n\n- It is not suitable for transactional communication. (Say if an OTP request needs to be done within 20 seconds, it is not applicable to go with, “as and when up” approach).\n Solution: Before making any transactional call, verify status of receiver and make a REST/gRPC API call. No other go, :-(\n- It is slightly complex to implement watcher.\n Solution: My next blog will address an easy way to implement CRD watcher, how to solve issues with watcher design and different ways to implement watching controller. Practise, you become perfect; Follow us, you become fantastic :-)\n\nSupport and follow us [**@gkGaneshR**](https://twitter.com/gkGaneshR) and [**@openebs**](https://twitter.com/openebs) to get instant updates.\n\nThanks [Kiran](https://twitter.com/kiranmova) for your valuable support. We at OpenEBS are always looking for help and feedback from Community. Please join us on [Slack](https://slack.openebs.io/) or comment on the [design doc](https://docs.google.com/document/d/1Q5W3uHktHa-vOm8oGp-3kpAQ3V1tvyk5AYmxxtf57Rg/edit?usp=sharing) and related [Pull Request](https://github.com/openebs/maya/pull/284).\n\n**Summary**\n\n- K8s CRD becomes a good alternative to REST/gRPC API for “push to perform” operations.\n- Few more implementation details will be covered in upcoming blogs — Follow us for updates.\n","notHasFeatureImage":true,"slug":"using-kubernetes-custom-resources-for-microservices-ipc"},{"id":96,"title":"Litmus - Release a chaos monkey on your Kubernetes Stateful Workloads!","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"01-05-2018","tags":["Chaos","Kubernetes","Litmus","Software Testing","E2e","Chaos Engineering"],"excerpt":"If you are a Kubernetes Enthusiast and working on stateful workloads, you may be asking yourself","content":"\n**In this blog we quickly talk about what led us to build Litmus and to open source it.**\n\nIf you are a Kubernetes Enthusiast and working on stateful workloads, you may be asking yourself:\n\n“With all the options I have to run Kubernetes — the permutations are endless — how can I be sure that my particular mix of options works well end to end at keeping my data safe and accessible?”\n\nYou are not alone, as can be seen by the ever increasing conversations on Kubernetes sig-storage slack channel and other forums like Reddit or Twitter. To just pick a few conversations:\n\n![Is it really recommended to run stateful workloads like MySQL on Kubernetes?](https://cdn-images-1.medium.com/max/800/1*6VJXdgFpuwD-fUkEKPo0GA.png)\n\n[Is it really recommended to run stateful workloads like MySQL on Kubernetes?](https://www.reddit.com/r/kubernetes/comments/88fxdg/is_it_really_not_recommended_to_run_stateful/)\n\n![What are the storage solutions offered in Kubernetes today? Which one will suit my workload](https://cdn-images-1.medium.com/max/800/1*5s60fO7nzhZfC3SFNiY0gA.png)\n\n[What are the storage solutions offered in Kubernetes today? Which one will suit my workload](https://twitter.com/rothgar/status/978694465975083009)\n\nAnd say, you somehow have made the journey to explore different solutions out there, mostly referring to product documentation and blogs, how can you be sure that the solution will continue to work in your enterprise environment?\n\nAs enterprises move to DevOps and microservices, more and more of the infrastructure from policy engines through storage and everything in between such as DNS, tracing, logging and more are selected and operated by all in one teams. With this control and autonomy comes greater agility — and all too often, _stress_.\n\nMeanwhile, infrastructure vendors and projects are also (we know first hand) challenged to keep their end-to-end (e2e) and chaos engineering frameworks updated with the ever-increasing permutations of deployment scenarios. Kubernetes itself is changing, new providers emerge every day, workloads are changing, and all of it is increasingly simple to adopt and deploy. As a storage solution provider we simply cannot have the resulting explosion of “corner cases” go untested.\n\nThe solution providers can go one step ahead to open source their project, but it still doesn’t help the users to ensure that the selected Kubernetes stack works in their highly distributed and agile environments and they are not called to fight fires at 3 AM.\n\n![Fire-fighting production issues !!](https://cdn-images-1.medium.com/max/800/0*qX8CliW_E3gKMURn.)\n\n“What’s a person to do? Test, test, release the chaos monkeys, and test again!”\n\nThankfully, Kubernetes and containerization and Go and some software engineering we’re happy to share make it much easier to provide end to end validation in real world conditions !\n\n#### So — What is Litmus?\n\n**_Litmus is a community for e-2-e testing and chaos engineering for Kubernetes, focusing on stateful workloads._**\n\nThe primary objective of Litmus is to ensure a consistent and reliable behavior of Kubernetes for various persistent workloads and to catch hard-to-test bugs and unacceptable behaviors before users do. Litmus can detect many more real-world issues than relatively simple issues identified by unit and integration tests.\n\nLitmus can also be used to determine if a given Kubernetes deployment is suitable for stateful workloads. While Litmus tests and metrics were developed initially to test the resilience of container attached storage from OpenEBS and others — we realized that the use cases are broader and overall system resilience can be characterized, which is a major reason we are open sourcing our efforts and putting the time into starting the Litmus community.\n\nLitmus tests range from initial setup and configuration validation to deploying and running persistent workloads under various conditions and failures.\n\n_What sets Litmus apart is not just its intent of being an end to end testing framework that can be embedded into any CI/CD pipeline, but the ease with which different teams from product developers to customers can contribute to the tests. Litmus allows for defining scenarios using native language specifications (English !!) OR a set of easy-to-define/understand YAML templates which are internally converted into test scripts, with a simple Kubernetes manifest as the end-product._\n\nHere is a simple test, defined in plain English:\n\n![Simple test in plain english](https://cdn-images-1.medium.com/max/800/0*ar6cYX2rEJ7Nh_G2.)\n\n## How to get involved with Litmus?\n\nFirst, it might be useful to understand the basic pieces of Litmus. Litmus has the following major components:\n\n![Litmus: High level architecture](https://cdn-images-1.medium.com/max/800/1*CdBbpkSilx3aJnZA3tiAjQ.png)\n\n- **Deployments** that help in setting up different types of Kubernetes Clusters like on-premise, cloud, OpenShift, etc. The default is that the deployments provision and configure OpenEBS storage, however, these deployments are easily extended to support other storage and we are happy to help any user or storage vendor to build additional deployments.\n- **Facilitators** for test execution that aid: defining and running test suites, capturing logs and generating reports about the test runs, fault/error injection tools that help to perform chaos tests, examples that demonstrate how to integrate these test pipelines with Slack notifications\n- **Test modules** that are triggered from within a Kubernetes cluster. Think of these as containerized tests. For instance, the **_mysql-client_** can be launched as a pod to validate MySQL resiliency while the underlying nodes and the connected storage are subjected to chaos engineering.\n- **Tests** that themselves are written in easy to understand formats, either in plain English (thanks [Godog](https://github.com/DATA-DOG/godog)!) or in Ansible Playbooks. These tests primarily interact with the Kubernetes cluster via **_kubectl_** making them highly portable.\n\nLitmus can be used to test a given workload in a variety of Kubernetes environments, for example, a developer minikube or a GKE cluster with a specific storage solution or as a part of a full-fledged CI setup.\n\nLitmus is early and needs all the help you can provide to have it cover the ever-growing Kubernetes landscape. Checkout the [Litmus Project](https://github.com/openebs/litmus) on Github for more details or if you are at KubeCon EU, please join us for the talk this Friday on [End to End testing with Kubectl](https://kccnceu18.sched.com/event/DqwD/using-kubectl-to-run-your-end-to-end-tests-amit-kumar-das-uday-kiran-mayadata-intermediate-skill-level) to learn more about how we have built Litmus and a quic\n\n#### Conclusion:\n\nPlease welcome Litmus into the world! We’re pretty sure it addresses a set of needs being felt by everyone from developers and operators to service providers and cloud native open source projects such as OpenEBS. With Litmus we use microservices and containers and Kubernetes to test, validate and characterize environments end to end. Your feedback is welcome and needed. Thanks for reading!\n","notHasFeatureImage":false,"slug":"litmus---release-a-chaos-monkey-on-your-kubernetes-stateful-workloads!"},{"id":97,"title":"How do I create an OpenEBS storage pool on Google Persistent Disk","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"13-04-2018","tags":["Docker","OpenEBS","Solutions","Google Cloud Platform","Kubernetes"],"excerpt":"The OpenEBS volume replicas, which are the actual backend storage units of the OpenEBS iSCSI target currently store the data in a hostPath on the Kubernetes nodes.","content":"\nThis article belongs to #HowDoI series on Kubernetes and OpenEBS.\n\nThe OpenEBS volume replicas, which are the actual backend storage units of the OpenEBS iSCSI target currently store the data in a hostPath on the Kubernetes nodes. By default, a folder with the volume (PV) name is created on the root filesystem, in a parent directory (/var/openebs) & bind mounted into the container during the replica pod instantiation. This parent directory (also created if not already available), which is basically a persistent path to hold the individual volumes is referred to as a **_Storage Pool_**.\n\nNote: The notion of the storage pool described above is specific to the current default storage engine,i.e., Jiva. Future releases may see availability of additional storage-engines which can consume block devices instead of hostdir to create storage pools\n\nFor various reasons, it may be desirable to create this storage pool on an external disk (GPD, EBS, SAN) mounted into specific locations on the Kubernetes nodes. This is facilitated by the **OpenEBS storage pool policy**, which defines the storage pool as a **_Kubernetes Custom Resource_** with the persistent path as an attribute.\n\nThis blog will focus on the steps to be followed to create the OpenEBS PV on Google Persistent Disks (GPD).\n\n## PRE-REQUISITES\n\n- 3-Node GKE cluster with the OpenEBS Operator installed.\n- 3-Google Persistent Disks, one attached to each node of the cluster.This can be done using the **_gcloud compute disks create_** & **_gcloud compute instances attach-disk_** commands (Refer for console steps: [https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk](https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk))\n\n### STEP-1: Format the GPDs & Mount into desired path\n\nOn each node, perform the following actions :\n\n- Switch to root user _sudo su –_\n- Identify GPD attached _fdisk -l_\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# fdisk -l\n Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors\n Units: sectors of 1 \\* 512 = 512 bytes\n Sector size (logical/physical): 512 bytes / 4096 bytes\n I/O size (minimum/optimal): 4096 bytes / 4096 bytes\n Disklabel type: dos\n Disk identifier: 0x635eaac1\n\n Device Boot Start End Sectors Size Id Type\n /dev/sda1 \\* 2048 209715166 209713119 100G 83 Linux\n\n Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors\n Units: sectors of 1 \\* 512 = 512 bytes\n Sector size (logical/physical): 512 bytes / 4096 bytes\n I/O size (minimum/optimal): 4096 bytes / 4096 bytes\n ```\n\n- Format the disk with, say ext4 fs (_mkfs.ext4 /dev/sd<>)_\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mkfs.ext4 /dev/sdb\n mke2fs 1.42.13 (17-May-2015)\n /dev/sdb contains a ext4 file system\n last mounted on /openebs on Fri Apr 13 05:03:42 2018\n Proceed anyway? (y,n) y\n Discarding device blocks: done\n Creating filesystem with 2621440 4k blocks and 655360 inodes\n Filesystem UUID: 87d36681-d5f3-4169-b7fc-1f2f95bd527e\n Superblock backups stored on blocks:\n 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632\n\n Allocating group tables: done\n Writing inode tables: done\n Creating journal (32768 blocks): done\n Writing superblocks and filesystem accounting information: done\n ```\n\n- Mount the disk into desired mount point (_mount -o sync /dev/sd<> /mnt/openebs_)\n\n ```\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mount -o sync /dev/sdb /mnt/openebs/\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# mount | grep openebs\n /dev/sdb on /mnt/openebs type ext4 (rw,relatime,sync,data=ordered)\n ```\n\n### STEP-2 : Create a storage pool custom resource\n\n- Construct a storage pool resource specification as shown below & apply it (Note that the custom resource definition for the storage pool is already applied as part of the operator install)\n\n ```\n apiVersion: openebs.io/v1alpha1\n kind: StoragePool\n metadata:\n name: sp-mntdir\n type: hostdir\n spec:\n path: \"/mnt/openebs\"\n ```\n\n### STEP-3 : Refer the storage pool in a custom storage class\n\n```\n---\napiVersion: storage.k8s.io/v1\nkind: StorageClass\nmetadata:\n name: openebs-custom\nprovisioner: openebs.io/provisioner-iscsi\nparameters:\n openebs.io/storage-pool: \"sp-mntdir\"\n openebs.io/jiva-replica-count: \"3\"\n openebs.io/volume-monitor: \"true\"\n openebs.io/capacity: 5G\n```\n\n### STEP-4 : Use the custom storage class in an application’s PVC spec\n\n```\n---\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: demo-vol1-claim\nspec:\n storageClassName: openebs-custom\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 5G\n```\n\n### STEP-5 : Confirm volume is created on the storage pool\n\n- Once the OpenEBS PV is created (_kubectl get pv, kubectl get pods_), list the contents of the custom persistent path mentioned in the storage pool custom resource. It should contain a folder with the PV name consisting of the sparse files (disk image files)\n\n ```\n karthik_s@strong-eon-153112:~$ kubectl get pv\n NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\n pvc-556e7ab7-3ed9-11e8-8e6a-42010a800216 5G RWO Delete Bound default/demo-vol1-claim openebs-custom 59m\n\n root@gke-oebs-staging-default-pool-7cc7e313-0xs4:~# ls /mnt/openebs/\n lost+found pvc-556e7ab7-3ed9-11e8-8e6a-42010a800216\n ```\n\n### GOTCHAS !!\n\n_Issue_: GPDs are detached in the event of a) Cluster resize (downscale/upscale) , b) upgrades & c) VM halts\n\n- No options to add “additional disks” during cluster creation\n- Instance templates are “immutable”, disks have to be added to instances separately\n\n_Workaround_: Perform a manual re-attach in above situations (Enlarged root disks are an option, but generally not recommended)\n","notHasFeatureImage":false,"slug":"how-do-i-create-an-openebs-storage-pool-on-google-persistent-disk"},{"id":98,"title":"Using chaoskube with OpenEBS.","author":"Sudarshan Darga","author_info":"Senior Software Engineer at MayaData","date":"12-04-2018","tags":["OpenEBS","Chaos Engineering","Kubernetes","Solutions"],"excerpt":"Chaos Engineering is the discipline of proving the reliability of any system by causing “chaos”. The word ‘Chaos’ means the state of confusion or failure caused due to unexpected reason.","content":"\n**Chaos Engineering** is the discipline of proving the reliability of any system by causing “chaos”. The word ‘Chaos’ means the state of confusion or failure caused due to unexpected reason.\n\n### Failures can be caused due to:\n\n- Power outages.\n- Software bugs.\n- Human Error.\n\n### Since failure is unavoidable.\n\n- Why not deliberately introduce failure to ensure the system can deal with the failure?\n- Chaoskube is one such tool, which can be used to introduce pod failures on Kubernetes Cluster.\n\n### Overview of Chaoskube:\n\n- Chaoskube is an open source Chaos Testing tool.\n- Written in GO language.\n- Can induce pod/controller failures on K8s Cluster.\n- Can kill pods by specifying the labels, namespaces.\n- Simple and easy to run.\n\n## Setup Chaoskube infrastructure on 3 node Kubernetes Cluster:\n\n```\nubuntu@kubemaster-01:~$ kubectl get nodes\nNAME STATUS ROLES AGE VERSION\nkubemaster-01 Ready master 2d v1.8.8\nkubeminion-01 Ready 2d v1.8.8\nkubeminion-02 Ready 2d v1.8.8\nkubeminion-03 Ready 2d v1.8.8\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/e2e/ansible/playbooks/resiliency/test-ctrl-failure/chaoskube.yaml\ndeployment \"chaoskube\" created\nserviceaccount \"chaoskube\" created\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/openebs/master/e2e/ansible/playbooks/resiliency/test-ctrl-failure/rbac.yaml\nclusterrole \"chaoskube\" created\nclusterrolebinding \"chaoskube\" created\nubuntu@kubemaster-01:~$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\nchaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 32s\nmaya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\nopenebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\n```\n\n## Deploy Percona application on OpenEBS volume with Liveness probe:\n\n```\nubuntu@kubemaster-01:~$ kubectl apply -f https://raw.githubusercontent.com/openebs/elves/master/e2e/percona-liveness/percona.yaml\ndeployment \"percona\" created\npersistentvolumeclaim \"demo-vol1-claim\" created\nservice \"percona-mysql\" created\nubuntu@kubemaster-01:~$ kubectl create configmap sqltest https://raw.githubusercontent.com/openebs/elves/master/e2e/percona-liveness/sql-test.sh\nconfigmap \"sqltest\" created\nubuntu@kubemaster-01:~$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\nchaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 6m\nmaya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\nopenebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\npercona-85b8997987-dg6jm 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-vd8t5 2/2 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-64rv5 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-b5v25 1/1 Running 0 1m\npvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-gs69w 1/1 Running 0 1m\n```\n\n## Induce controller failure using Chaoskube:\n\n- Induce failure on pod with label ‘openebs/controller=jiva-controller’ for duration of 60 seconds with interval of 20 seconds, which means it will induce controller pod failure for every 20 seconds for 3 times.\n\n ```\n kubectl exec chaoskube-55fc8f5f6d-tb6hj -- timeout -t 60 chaoskube --labels 'openebs/controller=jiva-controller' --no-dry-run --interval=20s --debug\n ```\n\n- Output should look something like this:\n\n ```\n time=\"2018-04-13T09:50:36Z\" level=info msg=\"terminating pod\" name=pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-vd8t5 namespace=default\n time=\"2018-04-13T09:50:36Z\" level=debug msg=sleeping duration=20s\n ```\n\n- Lets observe the failure induces by watching `kubectl get pods` for every 2 seconds.\n\n ```\n Every 2.0s: kubectl get pods Fri Apr 13 09:55:35 2018\n NAME READY STATUS RESTARTS AGE\n chaoskube-55fc8f5f6d-tb6hj 1/1 Running 0 16m\n maya-apiserver-69f9db69-b9qxk 1/1 Running 0 2d\n openebs-provisioner-77cb47986c-w6wbz 1/1 Running 1 2d\n percona-85b8997987-dg6jm 1/1 Running 7 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-ctrl-6fcb879bdb-nh5fk 2/2 Running 0 15s\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-64rv5 1/1 Running 0 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-b5v25 1/1 Running 0 12m\n pvc-0f07b9ae-3eff-11e8-8f7e-02b983f0a4db-rep-5df559c66c-gs69w 1/1 Running 0 12m\n ```\n\n- Observe that percona application pod with liveness probe is still running after inducing openebs controller pod failure using chaoskube. Hence, the system is reliable after causing ‘Chaos’.\n\n### Reference Links:\n\n- [https://github.com/linki/chaoskube](https://github.com/linki/chaoskube)\n- [https://openebs.io/docs](/docs)\n","notHasFeatureImage":false,"slug":"using-chaoskube-with-openebs."},{"id":99,"title":"How do I pin the OpenEBS Replica Pod(s) to the Kubernetes Nodes where they were scheduled?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"26-03-2018","tags":["Howdoi","Kubernetes","OpenEBS","Storage","Solutions","Tutorials"],"excerpt":"A OpenEBS Volume comprises of a Controller pod and one or more Replica pod(s). Controller pod (also known as a Target pod) is the one to which the application can make an iSCSI connection.","content":"\nThis article belongs to #HowDoI series on Kubernetes and OpenEBS.\n\nA OpenEBS Volume comprises of a Controller pod and one or more Replica pod(s). Controller pod (also known as a Target pod) is the one to which the application can make an iSCSI connection. The Replica pods are the ones that access the underlying disk resources for storing the data.\n\n**Use Case #1: In my Kubernetes cluster, [_OpenEBS volume pods are scheduled on appropriate nodes_](/blog/how-do-i-configure-openebs-to-use-storage-on-specific-kubernetes-nodes). This is all fine till the cluster experiences a disruption due to network partition. Kubernetes tries to evict & re-schedule these volume pods into newer nodes that does not have the underlying data. This results in volume getting into offline state. I want the OpenEBS volume pods to stick to the nodes they were originally placed.**\n\n**Solution**: Patch the Replica deployment with **nodeAffinity** property\n\nAs per Kubernetes docs, nodeAffinity allows you to constrain which nodes your pod is eligible to be scheduled on. It is based on labels on the node.\n\nThere are currently two types of node affinity:\n\n– `requiredDuringSchedulingIgnoredDuringExecution` &\n\n– `preferredDuringSchedulingIgnoredDuringExecution`\n\nThese node affinity types can be thought of “_hard_” vs. “_soft_” affinity respectively.\n\n- _Hard_ affinity states that pod will be scheduled only if the conditions are met.\n- _Soft_ affinity implies Kubernetes will make a best effort but the affinity may not be guaranteed.\n\nWe shall make use of `hard affinity` as this fits perfectly to the needs of Replica deployment.\n\nSteps required to patch the Replica deployment are summarised below:\n\n**Step 1**:- Replica pod(s) gets scheduled by Kubernetes default scheduler (via OpenEBS provisioner — a dynamic Kubernetes storage provisioner)\n\n**Step 2**:- Wait till replica pod(s) get into `Running` state\n\n**Step 3**:- Operator determines the node(s) on which the replica pod(s) are scheduled\n\n**Step 4**:- Replica deployment is patched with nodeAffinity\n\n```bash\n# REPLACE WITH ACTUAL NAMESPACE\n# REPLACE WITH ACTUAL PV NAME\n# TAKE A NOTE OF THE NODE NAME(S) TO BE USED IN THE PATCH.YAML\n\nkubectl get po -n \\\n -o=custom-columns=NAME:metadata.name,NODE:spec.nodeName,STATUS:status.phase \\\n | grep -E 'NAME|-rep'\n```\n\n```bash\n$ cat replica_patch.yaml\n```\n\n```yaml\nspec:\n template:\n spec:\n affinity:\n nodeAffinity:\n requiredDuringSchedulingIgnoredDuringExecution:\n nodeSelectorTerms:\n - matchExpressions:\n - key: kubernetes.io/hostname\n operator: In\n values:\n - nodename_where_replica_pod_1_got_scheduled\n - nodename_where_replica_pod_2_got_scheduled\n - nodename_where_replica_pod_3_got_scheduled\n```\n\n```bash\n# REPLACE WITH ACTUAL PV NAME\n\nkubectl patch deployment -rep \\\n -p \"$(cat replica_patch.yaml)\"\n```\n\n```bash\n# VERIFY IF PODs ARE BACK TO `Running` AFTER PATCH\n# REPLACE WITH ACTUAL NAMESPACE\n# REPLACE WITH ACTUAL PV NAME\n\nkubectl get po -n \\\n | grep -E 'NAME|-rep'\n```\n\nLearn more about nodeAffinity from Kubernetes docs at [https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity).\n\nIf you want to understand more on kubectl patch operation, then go through [https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/](https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/).\n","notHasFeatureImage":true,"slug":"how-do-i-pin-the-openebs-replica-pod(s)-to-the-kubernetes-nodes-where-they-were-scheduled?"},{"id":100,"title":"How do I configure OpenEBS to use storage on specific Kubernetes nodes?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"18-03-2018","tags":["Howdoi","OpenEBS","Solutions","Kubernetes","Tutorials"],"excerpt":"A OpenEBS Volume comprises of a Target pod and Replica pod(s). There can be one or more Replica pods. The Replica pods are the ones that access the underlying disk resources for storing the data.","content":"\nThis article belongs to **#HowDoI** series on [**Kubernetes**](https://kubernetes.io/) and [**OpenEBS**](https://openebs.io/).\n\n**Note: The approach mentioned in this article applies for OpenEBS version 6.0 or below. One can refer to this [link](https://github.com/openebs/community/pull/20) for OpenEBS version 0.8.0 and above.**\n\nA OpenEBS Volume comprises of a Target pod and Replica pod(s). There can be one or more Replica pods. The Replica pods are the ones that access the underlying disk resources for storing the data.\n\n**Use Case #1: In my Kubernetes Cluster, I have certain nodes that have disks attached. I call these as Storage Nodes. I want the OpenEBS Volume Replica Pods to be scheduled on these Storage Nodes.**\n\n**_Solution:_** Use Kubernetes “taints & tolerations” feature.\n\nAs per Kubernetes docs, taints allow a node to repel a set of pods. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes.\n\n- You can apply `NoSchedule` & `NoExecute` taints to the node(s).\n- `NoSchedule` marks that the node should not schedule any pods that do not tolerate the taint.\n- `NoExecute` marks that the node should evict existing/running pods that do not tolerate this taint.\n- Tolerations are applied to pods, and allow the pods to get scheduled onto nodes with matching taints.\n- You need to set an ENV variable in maya API server Deployment specifications, which in turn ensures setting of above tolerations on the replica pods.\n- The ENV variable referred to here is `DEFAULT_REPLICA_NODE_TAINT_TOLERATION`\n\nFollowing are the instructions to do the same:\n\n````\n# Step 1  —  Taint the node(s)\n```bash\n# kubeminion-01 is the name of a Kubernetes node\n# The taint effects used here are `NoSchedule` and `NoExecute`\nkubectl taint nodes kubeminion-01 storage=ssd:NoSchedule storage=ssd:NoExecute\n```\n\n# Step 2  —  Maya API server should be deployed with below specs\n# This ensures the replica pods are set with appropriate tolerations\n```yaml\napiVersion: apps/v1beta1\nkind: Deployment\nmetadata:\n name: maya-apiserver\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n name: maya-apiserver\n spec:\n serviceAccountName: openebs-maya-operator\n containers:\n —  name: maya-apiserver\n imagePullPolicy: Always\n image: openebs/m-apiserver:0.5.3\n ports:\n —  containerPort: 5656\n env:\n —  name: DEFAULT_REPLICA_NODE_TAINT_TOLERATION\n value: storage=ssd:NoSchedule,storage=ssd:NoExecute\n```\n````\n\n**Use Case #2:** In my Kubernetes Cluster, I have certain nodes that have disks attached. I call these as Storage Nodes. I want the OpenEBS Volume Replica Pods to be scheduled on these Storage Nodes. In addition, I want a better utilization of these nodes by being able to schedule my application Pods on these nodes as well.\n\n**Solution:** Use Kubernetes taints & tolerations feature. You may also want to try with `nodeAffinity` to achieve this. However, this solution focuses on use of tolerations.\n\n- You need to make use of `PreferNoSchedule` as the taint effect.\n- This can be thought of as a _soft version_ of `NoSchedule`.\n- In other words the system tries to avoid placing a pod that does not tolerate the taint on the node, but it is not mandatory.\n\nFollowing are the instructions to do the same:\n\n````\n# Step 1  —  Taint the node(s)\n```bash\n# kubeminion-01 is the name of a Kubernetes node\n# The taint effect used here is `PreferNoSchedule` i.e. a soft version of `NoSchedule` \n# the system tries to avoid placing a pod that does not tolerate the taint on the node,\n# but it is not mandatory.\nkubectl taint nodes kubeminion-01 storage=ssd:PreferNoSchedule\n```\n\n# Step 2  —  Maya API server should be deployed with below specs\n# This ensures the replica pods are set with appropriate tolerations\n```yaml\napiVersion: apps/v1beta1\nkind: Deployment\nmetadata:\n name: maya-apiserver\n namespace: default\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n name: maya-apiserver\n spec:\n serviceAccountName: openebs-maya-operator\n containers:\n —  name: maya-apiserver\n imagePullPolicy: Always\n image: openebs/m-apiserver:0.5.3\n ports:\n —  containerPort: 5656\n env:\n — name: DEFAULT_REPLICA_NODE_TAINT_TOLERATION\n value: storage=ssd:PreferNoSchedule\n ```\n````\n\nIf you want to learn more on taints & tolerations, then go through [https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)\n\nI shall strive to put more such articles in future. Do let me know if you want any specific topics that I should explain.\n\n_Thanks to Kiran Mova._\n","notHasFeatureImage":true,"slug":"how-do-i-configure-openebs-to-use-storage-on-specific-kubernetes-nodes?"},{"id":101,"title":"Achieving native hyper convergence in Kubernetes","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"13-03-2018","tags":["Hyper Convergence","Kubernetes","Nutanix","OpenEBS","Persistent Storage"],"excerpt":"Hyper convergence has a lot of benefits — which is one reason it has become popular in the traditional infrastructure-centric world of virtual machines with proprietary vendors like Nutanix reaching prominence over the last several years.","content":"\nHyper convergence has a lot of benefits — which is one reason it has become popular in the traditional infrastructure-centric world of virtual machines with proprietary vendors like Nutanix reaching prominence over the last several years.\n\nKubernetes is nearly ready as a layer enabling hyper convergence, as the compute orchestration is extremely flexible and networking has moved to a largely containerized approach that leverages local resources in attached physical and virtual hosts.\n\nWhen it comes to storage, however, there are a few pieces that are missing. Once added to Kubernetes, these pieces will unlock a number of benefits to users of Kubernetes including better resource utilization, reduction of noisy neighbor phenomena, simpler management, isolation at the node level thereby reducing the potential blast radius of failures, and, perhaps most importantly, further ownership and management of relevant infrastructure per workload and per DevOps team.\n\nStorage management capabilities in Kubernetes have improved in the last couple of years. For example, there is now clarity around how to connect a stateful application to persistent storage. The constructs of persistent volume claim (PVC), persistent volume (PV), and storage class (SC) along with dynamic provisioners from vendors have clarified how to connect a pod to a storage volume. With these Kubernetes constructs, a large ecosystem of legacy storage found its way to be connected to application pods. Many vendors and open source projects are so excited about this connectivity to cloud native environments that they have taken to calling their traditional storage “[cloud native](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing)”.\n\nIn order to explain why new tools and constructs are needed to improve the management of storage media, let’s start by reviewing pod connectivity. Shown below is a pod connected to external storage through a dynamic provisioner interface.\n\n![Need for new tools and constructs in Kubernetes for managing disks](https://cdn-images-1.medium.com/max/800/1*zm4UFgEvTWesM2JoxQF9Cg.jpeg)\n\nIn addition, we show the Local PV construct connected to local disks whether spinning or solid state.\n\nCurrently the Local PV can manage just a single disk. In a typical hyper-converged solution, more disks would be involved for a given pod. In addition to this single disk limitation, the following are limitations or gaps in the local PV feature.\n\n- Add disks / detach disks to a pod currently requires the pod to be restarted\n- Control and access to the storage is itself is limited. There will be a number of benefits of enabling access at times to the level of NUMA and to allowing CPU cores to be attached to the storage pod; keep in mind that these days using all the cores itself can be a challenge and driving up utilization is one of the central attributes of hyper converged systems\n- Today there is no standard ability to share the underlying disk. This is particularly important as extremely fast, and relatively expensive, NVMe SSD devices are now readily available and if shared they could be used as a cache for multiple Persistent Volumes\n- Lack of any fault management capabilities. Kubernetes needs to be able to receive and manage storage faults such as failures of the underlying disk, changes to the latency of disk IO\n\nWith these limitations in mind, we summarize needed enhancements as:\n\n- Capability to pool the disks, and provide an interface to manage the pooling\n- Capability to monitor disks, identify faults, forward them to the appropriate receivers\n\nIn the drawing above, these requirements are loosely shown as “constructs and tools to manage local disks”. These disk related constructs and tools are largely meant to be managed by Kubernetes cluster administrators and DevOps administrators in a Kubernetes like way.\n\nIt may be easier to think about this using human personas. Let’s say a DevOps developer is interested in connecting a working storage volume to their workload. Meanwhile a DevOps admin wants to rely as much as possible on Kubernetes to deliver storage services.\n\nToday, DevOps admins are forced to turning to different storage solutions to create storage classes as opposed to having a generic way of writing solutions around creating storage classes in a k8s native way. The DevOps admin would love to have a native k8s way to create storage classes so that they can standardize on an approach irrespective of the underlying storage systems or even storage cloud services.\n\n![Using and constructing a storage class in Kubernetes](https://cdn-images-1.medium.com/max/800/1*17YT5-GR_JUXEq6qW2SD1A.jpeg)\n\n## Some thoughts on what these new disk related constructs and tools could be\n\nJust like storage connectivity issue is solved with a dynamic volume provisioner, we could introduce pool provisioners into Kubernetes.\n\n![Proposed disk related constructs and interfaces](https://cdn-images-1.medium.com/max/800/0*eM2LjKDvhbl62mjG.)\n\nAs shown above, DevOps administrators will have the required tools to design the storage policy decisions. A toolset called is created node-disk-manager to provision, monitor, and manage disks on the node. The disks are then grouped into pools by an interface called pool provisioner. The pool provisioner gives a generic set of APIs to consume the Kubernetes disk objects and create a storage technology specific pools such as OpenEBS cStorPool, OpenZFS zpool, GlusterPool etc.The advantage of representing the pools in native Kubernetes constructs is that Kubernetes native tools can be extended to manage these new constructs.\n\nWith these constructs, the end-to-end volume provisioning work flow could be depicted like below.\n\n![New proposed workflow for managing local disks and achieving true hyper convergence in Kubernetes](https://cdn-images-1.medium.com/max/800/1*9bAs7wOPNNGLxELpgP-4FA.jpeg)\n\n## Conclusion:\n\nKubernetes native constructs and tools to manage and monitor local disks will help move towards achieving true hyper-convergence. We have observed many users in the OpenEBS community are using OpenEBS dynamic provisioner seamlessly and requesting tools to manage the disks and storage pools. We are thinking of adding such tools to Kubernetes itself so that they are available to the larger community. A draft design proposal is available at We are looking forward to proposing and discussing these thoughts with K8S SIG leadership.\n\nWe are cooking up some draft proposals in the OpenEBS project before we take the to K8S SIG. Your feedback is appreciated and needed.\n\n### Disk Manager design:\n\n[https://docs.google.com/presentation/d/11GLg21x7G-nMTNw8aNIOhhjW\\_-eK19zSI9Xm-0jYHKs/edit?usp=sharing](https://docs.google.com/presentation/d/11GLg21x7G-nMTNw8aNIOhhjW_-eK19zSI9Xm-0jYHKs/edit?usp=sharing)\n\n[https://github.com/kmova/node-bot/blob/3cb83976a5392003d02275f8a94d1860257915f0/design/node-storage-management.md](https://github.com/kmova/node-bot/blob/3cb83976a5392003d02275f8a94d1860257915f0/design/node-storage-management.md)\n\n### Pool Provisioner design:\n\n[https://github.com/kmova/openebs/blob/35fb65540b17ad3da3df270ccc425c4ec417ca12/contribute/design/proposal-cstor-orchestration.md](https://github.com/kmova/openebs/blob/35fb65540b17ad3da3df270ccc425c4ec417ca12/contribute/design/proposal-cstor-orchestration.md)\n","notHasFeatureImage":true,"slug":"achieving-native-hyper-convergence-in-kubernetes"},{"id":102,"title":"The Mule and (the) Flash — going for a run?","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"26-02-2018","tags":["Kubernetes","Cloud Storage","Container"],"excerpt":"In this blog, I discuss why we are building an innovative approach to user IO for the purpose of containerized storage, in particular vhost. If you just want the code, take a look at https://github.com/openebs/vhost-user.","content":"\nIn this blog, I discuss why we are building an innovative approach to user IO for the purpose of containerized storage, in particular vhost. If you just want the code, take a look at [https://github.com/openebs/vhost-user](https://github.com/openebs/vhost-user).\n\nFirst off, as mentioned in previous blog articles, OpenEBS is not yet another distributed file system. Let’s reiterate the reasoning behind this:\n\n- As microservices typically require only a small (relatively) amount of storage, there is no need to build a scale-out storage system\n- As Direct-attached Storage (DAS), in particular, NVMe, is the fastest storage you can get, you want the workload and the controller to be local with respect to each other; this is true even with SSD cloud storage offerings like AWS EBS instances\n- As single NVMe devices can reach 450K IOPS per device or [more](https://www.prnewswire.com/news-releases/supermicro-delivers-groundbreaking-18-million-iops-of-storage-performance-in-new-2u-ultra-server-300508258.html) there is no longer any need to “scale out” to achieve high IOPS or low latency, in fact, scale-out adds latency as per the above argument\n\nFinally, distributed applications are complex by nature. When you are building microservices, you are in fact, developing a distributed application. It seems unwise to put one distributed application on top of the other (storage) and sleep well at night. All that work you’ve done limiting single points of failure in your application layer can be undone through the use of complex distributed storage.\n\nAnother fundamental aspect of OpenEBS is that it runs in user space. This too has, we like to believe, a significant advantage as it does not require you to [build a kernel module](https://github.com/portworx/px-fuse) and taint your kernel (in case of closed source) with out-of-tree code. But it does not stop there; if you want to move your data from cloud to cloud (c2c), you do not have to worry about kernel version mismatches or anything like that. User space is the new kernel — when it comes to IO.\n\nBut what about performance? Linus Torvals himself said some years ago that file systems in user space are nothing but [toys](https://www.phoronix.com/scan.php?page=news_item&px=OTYwMA). But, as it turns out, with these low latency SSDs and high-speed networking (100GbE) the kernel, in fact, has become the bottleneck!\n\n_“fuse works fine if the thing being exported is some random low-use interface to a fundamentally slow device.”_\n\nSo it seems that we have reached an impasse? The kernel appears to be the bottleneck, and user space implementations are just “toys.” Or have we? When you look into why IO in user space is slow, it’s mostly due to the inability to do DMA, the required context switches, and the copying in and out of data. What if we could avoid this? Also, as you may know, hardware is already causing a change in the way we do things — 3D XPoint™ next to NVMe. This can be seen by technologies applied in SPDK and others like FD.IO. As OpenEBS is storage in containers, we have started to work on what we call the IOC, or the IO Container using these technologies.\n\n**The IOC runs in user space and can do IO to the underlying hardware, bypassing the kernel altogether. It owns a set of resources (CPU, NICs, memory, and storage) and applies polling for IO instead of being interrupt driven.**\n\nWith 18-core desktop computers being available today, it’s hardly an issue to use a core or two dedicated for IO — in the user space.\n\nBecause the IOC exposes block devices, we need a way to connect these devices to the other containers. Luckily — the VM space solved that problem for us: v[host](http://www.spdk.io/doc/vhost.html). By reusing these approaches, we create high a speed connection between the IOC and the containerized storage controller without making a change to the applications.\n\n![IOC model with vhost user interface and VPP](/images/blog/ioc-model-with-vhost-user-interface-and-vpp.png)\n\nThe above picture tries to depict the situation on a single node. As the application sends its IO through a block protocol (the target), OpenEBS — through the shared vhost subsystem — sends the IO to the replica which applies storage logic to it. With storage logic, we mean things that allow OpenEBS to do Copy-on-write (COW), snapshots, clones, compression or whatever is required. Also, OpenEBS is starting to further leverage this architecture to alter data management parameters including replication and snapshot patterns and even lower level parameters as well as block size in those containers depending on the workload.\n\nThen finally, the IO is submitted again to the IOC where an adaptive polling algorithm waits for its completion. Note, that the target — replicates `n` copies to the other node(s) which is depicted with R(n). So instead of doing IO through the kernel, your application passes the IO to the IOC which takes care of completing the IO as fast as possible all from user space.\n\nWith this approach, we get the best of both worlds and are in fact capable of surpassing the performance you would get when doing the same in the kernel — hands down — while also providing per workload granularity of control.\n\n![User space outperforming the kernel](/images/blog/user-space-outperforming-the-kernel.png)\n\nAs you can see from the repository, the design is fairly straightforward and is intended to support both legacy workloads as well as those built for faster underlying storage. We welcome input and contributions from anyone.\n\nWhile the vHost work stands alone it is central to a new storage engine forthcoming in OpenEBS 0.6, code named ‘cStore’.\n\nWe would really like your input so please [open an issue](https://github.com/openebs/vhost-user/issues) or join us on Slack to discuss at [openebs-community.slack.com](http://openebs-community.slack.com/) or just contact me directly. I can be reached at Twitter at [@jeffrymolanus](https://twitter.com/jeffrymolanus)\n","notHasFeatureImage":false,"slug":"the-mule-and-(the)-flash-—-going-for-a-run?"},{"id":103,"title":"OpenEBS plus Red Hat OpenShift and StackPointCloud and IBM Cloud Private and….","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"05-02-2018","tags":["DevOps","Docker","Kubernetes","Updates","OpenEBS"],"excerpt":"This week we announced that our partnership with Red Hat is flourishing. We achieved their Primed level of certification for their OpenShift offerings and are seeing more and more users rely upon OpenShift and Kubernetes as a means to provide persistence to their workloads.","content":"\nThis week we [announced](https://www.prnewswire.com/news-releases/openebs-certified-with-red-hat-openshift-stackpointcloud-and-ibm-cloud-672729373.html) that our partnership with Red Hat is flourishing. We achieved their Primed level of certification for their OpenShift offerings and are seeing more and more users rely upon OpenShift and Kubernetes as a means to provide persistence to their workloads.\n\nThe alternative pattern, of course, is to connect to an external storage system. Solutions like Rook and others such as the CSI efforts of RexRay and many others enable the use of external storage.\n\nActually — so does OpenEBS :)\n\nOpenEBS can and often does use external storage underneath. With OpenEBS, however, every workload has its own storage controller(s) that themselves are easily orchestrated by Kubernetes and data is local by default. There are three main benefits to the OpenEBS containerized architecture that external only storage cannot address due to architectural limitations:\n\n- **The granularity of control** — with OpenEBS the storage controller interprets ever more individualized and extensive storage policies and makes them so for each workload. Because OpenEBS is a full system (or is becoming one :)), it offers far more control than centralized storage that itself has to address the sometimes competing needs of countless — hundreds — of workloads. Developer teams can take on storage knowing they are much less constrained than they are working with least common denominator external storage.\n- **No SPOF** — in an age in which chaos engineering is becoming more and more popular, the notion of a sacrosanct dependency that cannot itself be disrupted or the entire system crashes potentially into a non-recoverable state is anachronistic. Put more directly — shared scale-out storage is an anti-pattern for many. Blast Radius.\n- **Performance** — as storage heads, we too often likely focus on performance. However, OpenEBS does work with databases, and in some cases the speed at which you run those workloads directly translates into user experience and hence money. So the tax a scale out storage system puts on performance versus the insane and rapidly accelerating speed of direct attached is essential. Ironically, scale-out first arose in part to work around how slow local disk was; times have changed. If you are interested in performance, you’ll want to grab our cStore by the way which, as the name suggests, is written in C and does much else as well to build upon our inherently faster Container Attached Approach. Stay tuned…\n\nSo why Red Hat and why StackPointCloud?\n\nIn both cases, we see organizations that are doing an incredible job helping their target users adopt Kubernetes based orchestration. With Red Hat, we tend to see especially larger enterprises taking the approach. With StackPointCloud, there is a real mix of departmental level users at large organizations as well as countless start-ups. In both cases, our support of Helm charts for OpenEBS makes it trivial to spin OpenEBS up.\n\nWhile OpenEBS itself as not achieve 1.0 status, we are working hand in hand w/ partners to make sure users are succeeding in their use of OpenEBS for stateful workloads. There must now be at least hundreds of production proof of concept deployments ongoing.\n\nPlease get in touch via [Slack](https://join.slack.com/t/openebs-community/shared_invite/enQtMjQzMTg4NTcyNTY2LTJiMzVjYjA5ZDk3YmI4NjAxY2QyYmI3MTA1MmUxMTAzNTU0NTM5NTViOTIxMjA1NWQ4NzVmMTBiNjk0NDU1YzQ) or otherwise if you would like to spend a little time with us to discuss your use cases and, of course, if you are running OpenEBS and testing it out.\n","notHasFeatureImage":false,"slug":"openebs-plus-red-hat-openshift-and-stackpointcloud-and-ibm-cloud-private-and…."},{"id":104,"title":"How to install OpenEBS on OpenShift?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"22-01-2018","tags":["Kubernetes","Mongodb","Openshift","Percona","Solutions","OpenEBS"],"excerpt":"OpenShift Origin is the upstream community project used in all deployment models provided by Red Hat OpenShift such as OpenShift Online, OpenShift Dedicated, and OpenShift Container Platform.","content":"\n## What is OpenShift Origin?\n\n**OpenShift Origin** is the upstream community project used in all deployment models provided by Red Hat OpenShift such as **OpenShift Online**, **OpenShift Dedicated**, and **OpenShift Container Platform**. **Red Hat OpenShift** is an enterprise container application platform that is based on a core of **Docker** container packaging, **Kubernetes** container cluster management and the **OpenShift Origin** project itself.\n\nFirst, having more than one option sounds confusing, but they clearly differentiate from each other. Here is the summary of all available deployment models to start with OpenShift.\n\n- [**OpenShift Origin**](https://www.openshift.org/) is a distribution of Kubernetes optimized for continuous application development and multi-tenant deployment. Origin is open source and all source code for the Origin project is available under the Apache License v2.0 on GitHub. Website and documentation for the Origin project are under [www.openshift.org](https://www.openshift.org/). It is completely free, you can deploy Origin on baremetal, in a VM or on a cloud. This is the option I will focus on this article.\n- [**OpenShift Online**](https://manage.openshift.com/) is Red Hat’s public cloud application development and hosting service. Starter plan is free to use and includes 1 project, 1GiB memory, 1GiB terminating memory and 1GiB storage. Pro Plan costs $50/month and includes 10 projects, 2GiB memory, 2GiB terminating memory and 2GiB storage. Details are available [here](https://www.openshift.com/pricing/index.html).\n- [**OpenShift Dedicated**](https://www.openshift.com/dedicated/index.html) is Red Hat’s managed private cluster offering, built around a core of application containers powered by Docker, with orchestration and management provided by Kubernetes, on a foundation of Red Hat Enterprise Linux. It’s available on the Amazon Web Services (AWS) and Google Cloud Platform (GCP) marketplaces. A complete OpenShift 3 cluster, configured for high availability (HA) with a minimum of 5 masters, infrastructure nodes and 4 application nodes managed by Red Hat costs $48k. Details are available [here](https://www.openshift.com/dedicated/index.html#pricing).\n- [**OpenShift Container Platform**](https://www.openshift.com/container-platform/index.html) (formerly OpenShift Enterprise) is Red Hat’s on-premise private PaaS product.\n\nIn this blog post, I will focus on configuring **OpenEBS** as a **persistent storage** option on the open-source self-managed **OpenShift Origin** and deploy a stateful workload both from CLI and custom catalog template using OpenEBS storage classes.\n\n## Prerequisites\n\n### Hardware\n\n- Minimum two nodes. Recommended four or more (Baremetal, VMs or cloud instances)\n\n### Software components used\n\n- [CentOS 7.x](https://www.centos.org/download/)\n- [OpenShift Origin 3.7+](https://github.com/openshift/origin)\n- [OpenShift-Ansible](https://github.com/openshift/openshift-ansible) (master branch used for installation)\n- [OpenEBS 0.5.1](https://openebs.io/)\n\n**Note:** Make sure the following package dependencies are installed: python, wget, git, net-tools, bind-utils, iptables-services, bridge-utils, bash-completion, kexec-tools, sos, psacct, docker-1.12.6, ansible, pyOpenSS, httpd-tool\n\n### Install OpenShift Origin\n\nFollow instructions from [OpenShift Origin Latest Documentation](https://docs.openshift.org/latest/welcome/index.html) to deploy a multi-node Origin cluster. If you are deploying it for the first time it may be a bit complicated. I plan to post my notes, steps on getting minimum requirements satisfied and have a successful deployment after this post.\n\n### Verify OpenShift Origin deployment\n\nExecute the following commands to verify successful installation.\n\n```\n# oc get nodes\n```\n\nNumber of nodes you see maybe different in your case, but status should looks similar to below showing nodes ready.\n\n```\n# oc get nodes\n NAME STATUS AGE VERSION\n oonode1 Ready 2d v1.7.6+a08f5eeb62\n oonode2 Ready 2d v1.7.6+a08f5eeb62\n oonode3 Ready 2d v1.7.6+a08f5eeb62\n oonode4 Ready 2d v1.7.6+a08f5eeb62\n```\n\n### Configure access permissions\n\nCreate a new admin user with cluster-admin role/permissions and assing password using the following commands:\n\n```\n# oc adm policy add-cluster-role-to-user cluster-admin admin — as=system:admin\n# htpasswd /etc/origin/master/htpasswd admin\n```\n\nLogin as the `admin` user and you will be using default project.\n\n```\n# oc login -u admin\n```\n\nOutput:\n\n```\n# oc login -u admin\n Authentication required for https://oonode1:8443 (openshift)\n Username: admin\n Password:\n Login successful.\nYou have access to the following projects and can switch between them with ‘oc project ’:\n* default\n kube-public\n kube-service-catalog\n kube-system\n logging\n management-infra\n openshift\n openshift-ansible-service-broker\n openshift-infra\n openshift-node\n openshift-template-service-broker\n openshift-web-console\nUsing project “default”.\n```\n\nProvide access to the host volumes which is needed by the OpenEBS volume replicas by updating the default security context (scc). If you miss this step your replicas will fail to deploy.\n\n```\n# oc edit scc restricted\n```\n\nNow set `allowHostDirVolumePlugin: true` and save changes. The file should look like below:\n\n```\n# Please edit the object below. Lines beginning with a ‘#’ will be ignored,\n# and an empty file will abort the edit. If an error occurs while saving this file will be\n# reopened with the relevant failures.\n#\nallowHostDirVolumePlugin: true\nallowHostIPC: false\nallowHostNetwork: false\nallowHostPID: false\nallowHostPorts: false\nallowPrivilegedContainer: false\nallowedCapabilities: []\nallowedFlexVolumes: []\napiVersion: v1\ndefaultAddCapabilities: []\nfsGroup:\ntype: MustRunAs\ngroups:\n— system:authenticated\nkind: SecurityContextConstraints\nmetadata:\nannotations:\nkubernetes.io/description: restricted denies access to all host features and requires\npods to be run with a UID, and SELinux context that are allocated to the namespace. This\nis the most restrictive SCC and it is used by default for authenticated users.\ncreationTimestamp: 2018–01–20T19:39:18Z\nname: restricted\nresourceVersion: “68274”\nselfLink: /api/v1/securitycontextconstraints/restricted\nuid: 9abddec5-fe19–11e7–8d06–005056873c08\npriority: null\nreadOnlyRootFilesystem: false\nrequiredDropCapabilities:\n— KILL\n— MKNOD\n— SETUID\n— SETGID\nrunAsUser:\ntype: MustRunAsRange\nseLinuxContext:\ntype: MustRunAs\nsupplementalGroups:\ntype: RunAsAny\nusers: []\nvolumes:\n— configMap\n— downwardAPI\n— emptyDir\n— hostPath\n— persistentVolumeClaim\n— projected\n— secret\n```\n\nSave changes.\n\n### Install OpenEBS on Origin\n\nThere are few easy ways to install OpenEBS. You can either apply the operator and storage classes direct from the URL or clone the repo and execute from the local copy. I prefer to clone a local copy, but i’ll also give you the other option if you prefer.\n\nClone the latest OpenEBS files and sample application specs using the below command on your OpenShift master node:\n\n```\n# git clone https://github.com/openebs/openebs.git\n# cd openebs/k8s\n```\n\nApply the file two yaml files below:\n\n```\n# oc apply -f openebs-operator.yaml\n# oc apply -f openebs-storageclasses.yaml\n```\n\nAlternative way — If you choose not to copy from the repo you can apply the yaml file direct from the URL below:\n\n```\noc apply -f https://openebs.github.io/charts/openebs-operator.yaml\n```\n\n### Verify OpenEBS deployment\n\nVerify that the OpenEBS provisioner and API server are created successfully and running.\n\n```\n# oc get deployments\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nmaya-apiserver 1 1 1 1 2d\nopenebs-provisioner 1 1 1 1 2d\n```\n\nCheck pods to confirm maya-apiserver and openebs-provisioner.\n\n```\n# oc get pods\nNAME READY STATUS RESTARTS AGE\ndocker-registry-1-b5r7t 1/1 Running 0 2d\nmaya-apiserver-3053842955-xbx8w 1/1 Running 0 2d\nopenebs-provisioner-2499455298–46brm 1/1 Running 0 2d\nregistry-console-1-mrpc9 1/1 Running 0 2d\nrouter-1-bf775 1/1 Running 3 2d\n```\n\nCheck services to confirm maya-apiserver exists.\n\n```\n# oc get service\nNAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE\ndocker-registry 172.30.113.229 5000/TCP 2d\nkubernetes 172.30.0.1 443/TCP,53/UDP,53/TCP 2d\nmaya-apiserver-service 172.30.17.113 5656/TCP 2d\nregistry-console 172.30.148.98 9000/TCP 2d\nrouter 172.30.229.239 80/TCP,443/TCP,1936/TCP 2d\n```\n\nCheck service accounts for openebs-maya-operator:\n\n```\n# oc get sa\nNAME SECRETS AGE\nbuilder 2 2d\ndefault 3 2d\ndeployer 2 2d\nopenebs-maya-operator 2 2d\nregistry 3 2d\nrouter 2 2d\n# oc get clusterrole openebs-maya-operator\n\\NAME\nopenebs-maya-operator\n# oc get clusterrolebindings openebs-maya-operator\nNAME ROLE USERS GROUPS SERVICE ACCOUNTS SUBJECTS\nopenebs-maya-operator /openebs-maya-operator default/openebs-maya-operator, default/default\n```\n\nAnd finally verify OpenEBS default storage classes.\n\n```\n# oc get sc\nNAME TYPE\nopenebs-cassandra openebs.io/provisioner-iscsi\nopenebs-es-data-sc openebs.io/provisioner-iscsi\nopenebs-jupyter openebs.io/provisioner-iscsi\nopenebs-kafka openebs.io/provisioner-iscsi\nopenebs-mongodb openebs.io/provisioner-iscsi\nopenebs-percona openebs.io/provisioner-iscsi\nopenebs-redis openebs.io/provisioner-iscsi\nopenebs-standalone openebs.io/provisioner-iscsi\nopenebs-standard openebs.io/provisioner-iscsi\nopenebs-zk openebs.io/provisioner-iscsi\n```\n\nAfter few easy steps we are now ready to deploy workloads on persistent storage provided by OpenEBS. I’ll cover both CLI and Catalog installation through the OpenShift Web Console.\n\n### Install Percona on OpenEBS using OC CLI\n\nUse OpenEBS as persistent storage for a Percona DB deployment by selecting the `openebs-percona` storage class in the persistent volume claim. I will use the sample file available in the openebs repo which I cloned locally in the previous steps.\n\nView the Percona deployment yaml:\n\n```\n# cd openebs/k8s/demo/percona\n# cat demo-percona-mysql-pvc.yaml\n— -\napiVersion: v1\nkind: Pod\nmetadata:\nname: percona\nlabels:\nname: percona\nspec:\ncontainers:\n— resources:\nlimits:\ncpu: 0.5\nname: percona\nimage: percona\nargs:\n— “ — ignore-db-dir”\n— “lost+found”\nenv:\n— name: MYSQL_ROOT_PASSWORD\nvalue: k8sDem0\nports:\n— containerPort: 3306\nname: percona\nvolumeMounts:\n— mountPath: /var/lib/mysql\nname: demo-vol1\nvolumes:\n— name: demo-vol1\npersistentVolumeClaim:\nclaimName: demo-vol1-claim\n— -\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\nname: demo-vol1-claim\nspec:\nstorageClassName: openebs-percona\naccessModes:\n— ReadWriteOnce\nresources:\nrequests:\nstorage: 5G\n```\n\nAs you can see in the YAML file above, `storageClassName` is set to `openebs-percona` which has 2 replicas.\n\nNow, apply the file:\n\n```\n# oc apply -f demo-percona-mysql-pvc.yaml\n```\n\nFinally, verify that Percona is deployed and OpenEBS controller and replica are running:\n\n```\n# oc get pods\nNAME READY STATUS RESTARTS AGE\ndocker-registry-1-b5r7t 1/1 Running 0 2d\nmaya-apiserver-3053842955-xbx8w 1/1 Running 0 2d\nopenebs-provisioner-2499455298–46brm 1/1 Running 0 2d\npercona-1378140207–5q2gb 1/1 Running 0 15mh\npvc-c7a24dc8-ffc7–11e7-a7cd-005056873c08-ctrl-1719480235-xf4t5 2/2 Running 0 15m\npvc-c7a24dc8-ffc7–11e7-a7cd-005056873c08-rep-1550141838-ldm59 1/1 Running 0 15m\n```\n\n### Install MongoDB on OpenEBS using the OpenShift Web Console\n\nLogin to the OpenShift Web Console using the admin credentials we have created earlier.\n\n![OpenShift Origin](https://cdn-images-1.medium.com/max/800/0*-IbP4t-ZgYZx4qh6.png)\n\nClick on **Add to Project** button and select **Import YAML / JSON**.\n\n![Add to project](https://cdn-images-1.medium.com/max/800/0*FEwbuF146LMi7Zsx.png)\n\nCopy the content of [https://raw.githubusercontent.com/openebs/openebs/master/k8s/openshift/examples/v3.7/db-templates/openebs-mongodb-persistent-template.json](https://raw.githubusercontent.com/openebs/openebs/master/k8s/openshift/examples/v3.7/db-templates/openebs-mongodb-persistent-template.json) file and paste into **Import YAML / JSON** window.\n\n![Import Yaml/JSON](https://cdn-images-1.medium.com/max/800/0*d6b0iSD6JG83ad-N.png)\n\nClick on **Create** button, select **Save template** and click **Continue**.\n\n![Add template](https://cdn-images-1.medium.com/max/800/0*14UvCpI6Gf-Q5Pd2.png)\n\nOn the **Template Configuration** window make sure Storage Class is `openebs-standard` and click on **Create**.\n\n![Template configuration](https://cdn-images-1.medium.com/max/800/0*l_agQ7YUPJnqvKkq.png)\n\n![Add template configuration](https://cdn-images-1.medium.com/max/800/0*K8665fQzu2nIGNZh.png)\n\n![Result](https://cdn-images-1.medium.com/max/800/0*E6Vp2d7hqBWtJpzm.png)\n\nYou have successfully deployed MongoDB on a persistent storage provided by OpenEBS.\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/how-to-install-openebs-on-openshift/)_._\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-openshift?"},{"id":105,"title":"Using OpenEBS as a Kubernetes persistent volume","author":"Jimmy Song","author_info":"Developer Advocate at Ant Financial, CNCF Ambassador, co-founder of ServiceMesher community, blog https://jimmysong.io","date":"10-01-2018","tags":["Kubernetes","OpenEBS","Docker","Cloud Native"],"excerpt":"OpenEBS is a containerized block storage written in Go for cloud native and other environments which make the data workloads more reliable in Kubernetes.","content":"\n[OpenEBS](https://www.openebs.io/) is a containerized block storage written in Go for cloud native and other environments which make the data workloads more reliable in Kubernetes.\n\nOpenEBS is open sourced by [MayaData](http://www.mayadata.io/) who is a professional containerized storage company formerly known as CloudByte. Their vision is to make data workloads easy to use in Kubernetes across clouds or on premise.\n\nWe know that [EBS](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://amazonaws-china.com/cn/ebs/&usg=ALkJrhhv8rYmHkvvZS_bPmr_Ca1Wj24SnA) (Elastic Block Storage) is available in AWS, persistent block storage for Amazon EC2 to meet the functional and performance requirements of the most demanding applications, and OpenEBS is its open source implementation.\n\n## Introduction\n\nWith OpenEBS, you can treat containers that have persistent data as you would any other common container. OpenEBS itself is also deployed through containers that support Kubernetes, Swarm, Mesos, Rancher orchestration scheduling, and storage services can be assigned to each pod, application, cluster, or container level, including:\n\n- Data persistence across nodes\n- Synchronize data across available zones and cloud vendors\n- Use commercial hardware and container engines to provide highly scalable block storage\n- Integration with the container orchestration engine, the developer’s application can automatically configure OpenEBS\n- Based on CloudByte’s container-based experience in BSD, we provide users with OpenEBS QoS assurance\n\n## Architecture\n\nThe OpenEBS storage controller itself runs in a container. OpenEBS Volume consists of one or more containers that run microservices. This storage controller function is based on a microservices architecture — the data for each volume is provided by its own set of containers, not by a single monolithic storage controller that provides control for multiple volumes at the same time To provide. This is the essential difference between OpenEBS and traditional storage devices.\n\nThe OpenEBS architecture can be divided into Data Plane (Data Plane) and Control Plane (Control Plane) in two parts:\n\n- Data Plane: Provides data storage for applications\n- Control Plane: Managing OpenEBS Volume Containers, which typically uses the functionality of container layout software\n\n## Data plane\n\nThe following figure shows the architecture of OpenEBS deployed on Kubernetes cluster. Among them, the yellow or orange part is the OpenEBS persistent storage volume, created by Kubernetes’ PVs, implemented using iSCSI, and stored on host nodes or in the cloud (such as EBS, GPD, etc.) depending on where your cluster is deployed. The OpenEBS volume is completely independent of the user’s application life cycle to manage, which is Kuberentes PV in the basic idea.\n\n![OpenEBS Cluster - Data Pane](/images/blog/openebs-data-plane.png)\n\nOpenEBS volumes provide persistent storage for containers with resiliency to system failures and faster access to storage, snapshots and backups. In addition, it provides mechanisms for monitoring usage and enforcing QoS policies.\n\nThe disk that stores the data is called the storage backend and can be a host directory, an attached block device, or a remote disk. Each OpenEBS volume contains an iSCSI target container (represented as openebs-vol1 in the previous figure) and one or more replica containers (openebs-vol1-R1 and openebs-vol1-R2).\n\nThe application pod accesses the storage through the iSCSI target container, which copies the data to all of its replicas. In the event of a node failure, the iSCSI target container starts from one of the remaining online nodes and provides data by connecting to the available replica containers.\n\n**Source**\n\nThe implementation of this section consists of two containers:\n\n- [openebs/jiva](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/jiva&usg=ALkJrhhhCfHb4LkQReHbpayqLJwjwdctgw) : storage control functions, including copy logic\n\n- [openebs/gotgt](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/gotgt&usg=ALkJrhgoXb10SL2TVf8_urB_TIfEVSDBxg) : iSCSI target features used by openebs/jiva\n\n## Control plane\n\nThe OpenEBS control plane is also known as maya. The purpose is to create a hyper-converged OpenEBS that is mounted on a container scheduling engine such as Kubernetes, Swarm, Nomad, etc. to extend the storage capabilities provided by a particular container orchestration system.\n\n![OpenEBS Cluster - Control Plane](/images/blog/openebs-control-plane.png)\n\nOpenEBS’s control plane is also based on microservices, and its services can be divided into the following sections:\n\nContainer layout plug-in, used to enhance the function of the strong container layout framework:\n\n- **Kubernetes Dynamic Configuration** : [openebs-provisioner](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/external-storage/tree/master/openebs&usg=ALkJrhjuOf_IBvwR0NC-g734l_p4Ia14hg)\n- **Kubernetes-dashboard** : [openebs-dashboard](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/dashboard&usg=ALkJrhigRmJSDzmVT_NRMupygPwAM5EX9g)\n- **Extended schema** : Kubernetes-based CRDs (custom resource defination) that store OpenEBS-related configuration data\n\nCluster services provide OpenEBS-specific storage intelligence such as:\n\n- **maya-apiserver** : Contains APIs for performing volume operations that translate requests into container-specific system-specific operations\n- **maya-mulebot** : Use the information collected to suggest optimized layout and event handling tips\n- **maya-connect** : Allows monitoring data to be uploaded to `maya-cloud` for further storage access mode analysis\n\nNode Services, which provide OpenEBS-specific storage intelligence that runs with kubelet, such as:\n\n- **maya-agent** : Includes storage management features\n\nBy using prometheus, heapster, grafana and jaegar for these services, you can add monitoring and tracking capabilities.\n\n**Source**\n\n- [openebs / maya](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/maya&usg=ALkJrhgksSLVDOSt9WRSnCdGdaf4nezkyQ) : All of the specific binary code (non-plugins) is stored in this repository, such as `maya-apiserver` , `maya-agent` , `maya-mulebot` , `maya-connect` , `mayactl` and more.\n- [openebs-dashboard](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/dashboard&usg=ALkJrhigRmJSDzmVT_NRMupygPwAM5EX9g) : A branch of the kubernetes-dashboard project that extends storage capabilities.\n- [openebs-provisioner](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=https://github.com/openebs/external-storage/tree/master/openebs&usg=ALkJrhjuOf_IBvwR0NC-g734l_p4Ia14hg) : The OpenEBS K8s Provisioner from the Kubernetes incubator project.\n\n## Install OpenEBS on Kubernetes\n\nBelow we will use the way to install OpenEBS operator, you need to make sure you have already installed iSCSI on your node before installation.\n\n## Prerequisites\n\nOpenEBS relies on iSCSI for storage management, so you need to make sure that you have OpenEBS installed on your cluster.\n\n**Note** : If you are using kubeadm, container-mounted kublet, it comes with iSCSI and does not need to be manually installed. For a kubelet installed directly on the bare metal in binary form, you need to install iSCSI yourself.\n\nThe iSCSI (Internet Small Computer System Interface) is a TCP / IP-based protocol used to establish and manage interconnections between IP storage devices, hosts and clients, and to create storage area networks (SANs ). The SAN makes it possible for the SCSI protocol to be used in high-speed data transmission networks, with block-level data transfer between multiple data storage networks. The SCSI architecture is based on C/S mode and is typically used in environments where devices are close to each other and these devices are connected by a SCSI bus.\n\nOpenEBS needs to use iSCSI as a storage protocol, and CentOS default does not have iSCSI installed, so we need to manually install.\n\nThere are two types of roles in iSCSI:\n\n- **target** : used to provide storage (server)\n- **initiator** : use the stored client (client)\n\nThe following figure in Kubernetes uses iSCSI architecture (Source: [http://rootfs.github.io/iSCSI-Kubernetes/)](https://translate.googleusercontent.com/translate_c?depth=1&hl=en&rurl=translate.google.co.in&sl=zh-CN&sp=nmt4&tl=en&u=http://rootfs.github.io/iSCSI-Kubernetes/%25EF%25BC%2589&usg=ALkJrhgk4iuBd1pHB1zGq6XKLwffkSGZew)\n\n![iSCSI-Kubernetes](/images/blog/iscsi-kubernetes.png)\n\nInstalling the iSCSI service is very simple, you do not need additional configuration, just start the service after installation.\n\nExecute the following command on each node node:\n\n```\nyum -y install iscsi-initiator-utils systemctl enable iscsid systemctl start iscsid\n```\n\n## Quick start\n\nRun the OpenEBS service using Operator:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-operator.yaml kubectl apply -f openebs-operator.yaml\n```\n\nUse the default or custom storageclass:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/openebs-storageclasses.yaml kubectl apply -f openebs-storageclasses.yaml\n```\n\nMirror used are:\n\n- openebs/m-apiserver: 0.5.1-RC1\n- openebs/openebs-k8s-provisioner: 0.5.1-RC2\n- openebs/jiva: 0.5.1-RC1\n- openebs/m-exporter: 0.5.0\n\n## Test\n\nLet’s use the Example from the official OpenEBS documentation to install the Jenkins test:\n\n```\nwget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/jenkins/jenkins.yml kubectl apply -f jenkins.yml\n```\n\nCheck PV and PVC\n\n```\n$ kubectl get pv\nNAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE\npvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0 5G RWO Delete Bound default/jenkins-claim openebs-standard 1h\n$ kubectl get pvc kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\njenkins-claim Bound pvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0 5G RWO openebs-standard 1h\n```\n\nView Jenkins pod:\n\n```\nEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 29m (x2 over 29m) default-scheduler PersistentVolumeClaim is not bound: \"jenkins-claim\" (repeated 3 times) Normal Scheduled 29m default-scheduler Successfully assigned jenkins-668dfbd847-vhg4c to 172.20.0.115 Normal SuccessfulMountVolume 29m kubelet, 172.20.0.115 MountVolume.SetUp succeeded for volume \"default-token-3l9f0\" Warning FailedMount 27m kubelet, 172.20.0.115 Unable to mount volumes for pod \"jenkins-668dfbd847-vhg4c_default(8e2ad467-f1e5-11e7-aa47-f4e9d49f8ed0)\": timeout expired waiting for volumes to attach/mount for pod \"default\"/\"jenkins-668dfbd847-vhg4c\". list of unattached/unmounted volumes=[jenkins-home] Warning FailedSync 27m kubelet, 172.20.0.115 Error syncing pod Normal SuccessfulMountVolume 26m kubelet, 172.20.0.115 MountVolume.SetUp succeeded for volume \"pvc-8e203e86-f1e5-11e7-aa47-f4e9d49f8ed0\" Normal Pulling 26m kubelet, 172.20.0.115 pulling image \"sz-pg-oam-docker-hub-001.tendcloud.com/library/jenkins:lts\" Normal Pulled 26m kubelet, 172.20.0.115 Successfully pulled image \"sz-pg-oam-docker-hub-001.tendcloud.com/library/jenkins:lts\" Normal Created 26m kubelet, 172.20.0.115 Created container Normal Started 26m kubelet, 172.20.0.115 Started container\n```\n\nStart up successful. The Jenkins configuration uses **NodePort** mode access and now accesses the NodePort of Jenkins service for any node in the cluster.\n\n## Reference\n\n- [OpenEBS Documentation](http://openebs.readthedocs.io/)\n- [CentOS 7.x 下配置 iSCSI 网络存储](http://blog.csdn.net/wh211212/article/details/52981305)\n- [Configure iSCSI Initiator](https://www.server-world.info/en/note?os=CentOS_7&p=iscsi&f=2)\n- [https://www.openebs.io/](https://www.openebs.io/)\n- [https://github.com/openebs/openebs](https://github.com/openebs/openebs)\n- [Data Scientists adopting tools and solutions that allow them to focus more on Data Science and less on the infrastructure around them](/blog/data-scientists-adopting-tools-and-solutions-that-allow-them-to-focus-more-on-data-science-and-less)\n- [RHEL7: Configure a system as either an iSCSI target or initiator that persistently mounts an iSCSI target.](https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistently/)\n\nOriginal page: [https://jimmysong.io/posts/using-openebs-as-kubernetes-persistent-volume/](https://jimmysong.io/posts/using-openebs-as-kubernetes-persistent-volume/)\n\nTranslated from Chinese to English by Google Translate\n","notHasFeatureImage":true,"slug":"using-openebs-as-a-kubernetes-persistent-volume"},{"id":106,"title":"In 2018 - IT dreams deferred finally achieved?","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"09-01-2018","tags":["Containerization","Docker","Kubernetes","OpenEBS","Storage"],"excerpt":"At MayaData, we believe we, and others are building the foundation for a much longer cycle of software-centric innovation thanks to proactively eliminating sources of lock-in.","content":"\n# **_…..Dreams deferred_**\n\nMany of us in the infrastructure business have been forced by experience to lower our expectations of what is possible. While we’ve all dreamed for decades of a world in which software just works — and delivers value where and how it is needed — we’ve been disappointed again and again.\n\nWe have seen open systems that, over time, became increasingly proprietary with Unix diverging into proprietary camps.\n\nWe’ve seen SQL go from a fascinating research project to a broadly deployed standard to, with the help of stored procedures and truly nefarious licensing, a source of lock-in dominated by one company and one eccentric multi-billionaire.\n\nWe’ve seen a vision of Java as a cross infrastructure abstraction layer bloom and wither.\n\nAnd of course, we’ve seen virtual machines offer the promise of cross infrastructure mobility only to fall prey to the rest of the stack and proprietary business models and incorrect levels of abstraction.\n\nOver time the result has been infrastructure by silos, with each silo — security, storage, networking and compute — dominated by proprietary solution providers that over time sought to provide the entire stack to drive up their sales, even if doing so meant increasing the friction for users seeking to combine best of breed solutions.\n\n# **_On the other hand…._**\n\nAll along technological progress has continued. Allowing for new possibilities.\n\nWe’ve even seen — finally — broadband make its way into the United States so that more and more we can cost effectively access the cloud (yes, the loss of net neutrality seems to put this at risk for at least consumers and new entrants).\n\nAnd intra data center networking has gotten insanely fast — which is crucial if we are to run workloads in a flexible manner.\n\nAnd arguably the best example of innovation at scale in modern business — Amazon — focusing on the right persona — the developer — and raising the bar massively for all of us in infrastructure.\n\nAnd perhaps most importantly, the Open Source community, which as Richard Stallman and others have pointed out predates the commercial software world and which some have called the world’s first social network, grew to become an undeniable force.\n\nAnd pulling all the positive forces together — DevOps and microservices. DevOps as a cultural movement and approach to building and running software at scale PLUS an emerging understanding of how to run systems via microservices as explained by the [12 factor approach](https://www.12factor.net/) and elsewhere led to countless examples of “software eating the world.”\n\n# **But …. What about lock-in?**\n\nSo as the above suggests, one theme in the story of innovation in IT over the years has been breakthrough technologies, and business models, enabling fundamentally better software delivered more easily to users. And as one approach came to predominate, proprietary approaches over time led to more “rent seeking”, where leading vendors extracted more value from their users and slowed their innovation. And this stagnation leads to pent up demand for better approaches — triggering the next cycle.\n\nWell — what about this time?\n\nKubernetes has emerged in part because it promises a world more free from lock-in to AWS and other clouds. Could it be that we have collectively learned enough from all the boom and bust cycles to know what is good for us?\n\nCould be — the signs are incredibly promising as all the cloud vendors and RedHat and Cloud Foundry and Docker and Mesos have all embraced Kubernetes as the standard control plane. This means that you are no longer locked-in by the control plane logic and should be able to move your applications from cloud to cloud and from on premise to off. Crucially — Kubernetes itself is open source and all the major vendors have pledged to not fork it; so it shouldn’t be _too_ bad to move from one vendor supporting Kubernetes to another.\n\n_…. but what about data?_ Without data mobility all you can move is the stateless components of your applications — provided you address having those components able to access your store of state.\n\n# **And your data remains largely locked-in**\n\n_Locked into_ proprietary vendors.\n\n_Locked into_ underlying systems that are sources of risk and that themselves are resolutely monolithic.\n\nI harken back to a speech Randy Bias gave at one of the OpenStorage summits I helped host back in 2010 about `blast radius`. The basic idea is that microservices dramatically reduce the blast radius of any single outage; conversely putting all your state in a shared storage system is, by comparison, an anti-pattern. When your shared storage dies or slows down unexpectedly perhaps due to a rebalancing, so does your entire environment. So much for being built for failure!\n\nS3 for non performant data and EBS for performant data have become defacto standards. They are easy, they “just work”, and — crucially — they put the responsibility for the configuration, care and feeding of state in the hands of the teams that also control the microservices.\n\nThe only problem is that it is _hard_ to move your data from these AWS services to other solutions without a lot of work that frankly software development teams don’t have the time or inclination to invest. I see the lock-in that results as the TBs pile up treated much as technical debt is treated — it is annoying and yet it is much less important than getting valuable capabilities in the hands of end users.\n\nAnd putting all your data in a scale-out software solution running on these clouds only makes the issue worse. Now you have the blast radius issue and you have your data stored in a solution that cannot be stretched across clouds. Two sources of lock-in and at least twice the effort!\n\nIt might be worth remembering that networking, security and compute are all becoming both infrastructure services delivered as services **to** today’s microservice environments and **are themselves also microservice based services**. Take a look at Project Calico for instance. Or at Kubernetes itself.\n\n**Nobody says — hey, Kubernetes is just a black box that sits to the side and so it needn’t be a bunch of microservices. But not storage. Storage somehow gets a pass. It gets to live with aged architectures and typically aged business models.**\n\n## Which raises the question: What if storage was itself delivered as microservices and orchestrated by Kubernetes?\n\nFor the purpose of this exercise, **assume** it were possible to make storage a set of capabilities delivered as microservices with the controller running on containers.\n\nYou’d probably agree that such an approach would have some benefits including:\n\n**Familiarity:**\n\n- If storage is delivered as microservices within Kubernetes then if you know how to run Kubernetes then you know how to run the storage.\n- Perhaps more importantly, you are familiar with the failure domain. You lose a storage controller — well, you just lost a stateless container that itself simply provides services and pointers towards the underlying data. Your data is always safe in multiple locations and your storage system itself is resilient (at least the way OpenEBS is architected with the use of atomic transactions).\n\n**Granularity:**\n\n- As mentioned above, the defacto standard approach to delivering storage is to use AWS itself with each team organized around one or more microservices having their own approach to EBS for performant storage and S3 for blobs of data.\n- Using a shared storage system runs counter to this approach and cuts these teams out of the loop. They are back to lobbying central IT as one of hundreds or even thousands of workloads with particular desires as to how storage should be configured. And, yes, those configurations matter. And, actually, they are impossible to get right. We’ve talked about that in the past including at Meet-ups: [https://www.slideshare.net/MattBaldwin3/containerized-storage-for-containers-why-what-and-how-openebs-works](https://www.slideshare.net/MattBaldwin3/containerized-storage-for-containers-why-what-and-how-openebs-works)\n\n![What move the data and configs next to the app](/images/blog/what-move-the-data-and-configs-next-to-the-app.png)\n\n**Performant:**\n\n- This being a storage blog, it is worth reiterating the point that shared storage is inherently less performant these days than direct attached or DAS. That is a fairly new reality. It used to be that DAS was really slow disk and the way to get IOPS was to stripe across a bunch of faster disks. That was a primary driver for shared storage. Imagine that — at one time CEPH would have been faster than the underlying hardware! How times have changed.\n- Our CTO, Jeffry Molanus does a good job walking through how the landscape of performance has changed why this and other changes now favor what we call “container attached storage”:\n- [https://openebs.io/blog/not-yet-another-distributed-storage-system](/blog/not-yet-another-distributed-storage-system)\n\n**Natively cross cloud — with the help of metadata and routing services:**\n\n- What is perhaps least well appreciated about the potential of treating storage as a service delivered via microservices is that, correctly engineered, this means that data itself can be served as a service in the background across underlying clouds.\n- The first prerequisite is that the controller itself runs in a container or set of containers.\n- The second prerequisite is that the controller performs its magic in the user space so that the container does not need to be a special build and so that the system can perform.\n- Third, there needs to be the management of metadata to see where the data is versus the workloads. Kubernetes can help here as it expands.\n# **TL;DR:**\n\nSo, in short, this time perhaps it really _is_ different.\n\nThis time we “won’t get fooled again” (gratuitous old guy music reference :)).\n\nThis time we _will_ address the sources of lock-in not just at the controller plane via Kubernetes but also at the data layer. And in so doing we will avoid ending the the cycle of innovation prematurely. Perhaps it goes without saying — only an open source solution like OpenEBS that is widely accepted and easy to fork if needed can help free us from the risk of cloud lock-in without adding yet another source of lock-in.\n\nAnd we can address lock-in while respecting and extending the patterns we know are working including: every team controlling their infrastructure themselves, the elimination of single points of failure (aka “storage blast radius”), and allowing Kubernetes to control more and more of the environment, leaving the developers to focus on capabilities that add value to their end users.\n\nIn short, at MayaData we believe we and others are building the foundation for a much longer cycle of software-centric innovation thanks to proactively eliminating sources of lock-in.\n\nPlease help this reality come true by providing us feedback on [OpenEBS](http://www.openebs.io/) and [MayaData](http://www.mayadata.io/) or see us on the Kubernetes storage SIG where we are trying to be helpful as well.\n","notHasFeatureImage":false,"slug":"in-2018---it-dreams-deferred-finally-achieved?"},{"id":107,"title":"Austin KubeCon — Persistent Storage Round-up and Looking beyond!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"08-01-2018","tags":["Kubecon","Kubernetes","Persistence","Storage Containers","Updates"],"excerpt":"Kubernetes Clusters are up and running at the push of a button or even better by talking to your favorite bot.","content":"\nTL;DR\n\nThis rather has become a long post as I drafted it and incorporated feedback from community members. So, in short:\n\n- 2017 saw Kubernetes being crowned as the de-facto container orchestration engine. And from the storage perspective, containerized storage makes its presence felt.\n- 2018 — The reign of Kubernetes continues. Containerized Storage gains momentum with renewed focus on manageability of persistence workloads\n\n—\n\nKubernetes Clusters are up and running at the [push of a button](https://twitter.com/muratkarslioglu/status/941399154714066944) or even better by [talking to your favorite bot](https://www.youtube.com/watch?v=07jq-5VbBVQ).\n\n![Push a button](https://cdn-images-1.medium.com/max/800/1*oz5esyJvsb5zBIaoyDKUeQ.png)\n\nsource: Containerized Storage for Containers — session at Kubernetes Meetup [https://t.co/tdQaOue5w8](https://t.co/tdQaOue5w8)\n\nBut just about a year ago when we started envisioning OpenEBS — Containerized Storage for Containers — to be orchestrated by Kubernetes, setting up a cluster took a good three days. Phenomenal progress by the Kubernetes Community in 2017 — from Kubernetes — the Hard-Way to making it Child’s play!\n\nIf you were at KubeCon, you would have definitely been caught up in or at least glimpsed the Euphoria around Kubernetes. Kubernetes, almost feels like Noah’s Ark right now — you are either in or wait to perish. A little exaggerated, I know, but only a little.\n\n_Every Technology and Cloud Service Provider are now providing or planning to provide a container service using Kubernetes and almost every infrastructure provider is looking at putting themselves on the map of Kubernetes._\n\n![Cloud Native Landscape](https://cdn-images-1.medium.com/max/800/1*YJIF6xBEPL1WpVgOK4VV0Q.png)\n\n[https://raw.githubusercontent.com/cncf/landscape/master/landscape/CloudNativeLandscape_latest.png](https://raw.githubusercontent.com/cncf/landscape/master/landscape/CloudNativeLandscape_latest.png)\n\nAnd why not!\n\nKubernetes has reached the level of maturity that can be used with ease in controlled environments and at the same time, has gained tremendous strength from a community that is not afraid to re-engineer or re-architect the core components. The popularity of Kubernetes is enabling many meta-kubernetes projects like — kubespray, stackpointcloud, kubeless, heptio ksonnet, heptio ark, etc. And with these new projects and possibilities, many _Kubernetes — blue ocean — companies are on the rise!_\n\n—\n\nI am very bullish that Kubernetes is that magical ingredient that will renew the focus on HumanOps!\n\nOne inevitable aspect of being an infrastructure operations admin is to be prepared for smooth operations, scaling, maintaining and recovering from faults and disasters — which usually tend to put a lot of unwarranted pressure on the admins when dealing with their own management (business impact) and the vendors whose technology they used to build a “black-box” infrastructures. A “black-box” infrastructure that doesn’t comply with what they were told and assumed would do — and involves talking to people building those black-boxes often crossing company borders, leading into non-technical calls about blaming who is at fault. Such stressful conditions also exist within an organization where there is a crunch of resources.\n\nKubernetes and the meta-kubernetes projects are helping administrators build — what I call “white-box” infrastructures. Often professed and hardly-prevailed aspect of Infrastructure is the [_HumanOps_](https://blog.serverdensity.com/humanops/) _— \"_ technology affects the well being of humans just as humans affect the reliable operation of technology *\"* — which can be achieved by building “white-box” infrastructures that are easy to operate and reduce the dependency on specialists that tend to be over-worked in an organization. _The “white-box” infrastructures are built with API-driven Open Source Micro-Services._\n\nThe key to the widespread adoption of Kubernetes in such a short time is the inclusive nature of it, which was well captured by this slide from the KubeCon:\n\n![Extensibility](https://cdn-images-1.medium.com/max/800/1*IXods_RnXRco2z7UcngePw.png)\n\n[https://schd.ws/hosted_files/kccncna17/ac/KubeCon_2017\\_-_Kernels_and_Distros.pdf](https://schd.ws/hosted_files/kccncna17/ac/KubeCon_2017_-_Kernels_and_Distros.pdf)\n\n### **_Kubernetes is more than an orchestration engine → It is the new kernel for building clustered infrastructures._**\n\nI consider this shift towards making Kubernetes a Kernel that can be extended by custom solutions that can be downloaded and installed as a true enabler for driving innovation — which is inline with the Psyche of keeping “Community First and Company Next!”.\n\n—\n\nThis past year, saw the rise in user awareness for securing Containers. Different teams started tackling this issue from different perspectives — from providing secure container runtime like Kata Containers to using different types of Service Meshes to better access control and more.\n\nService Mesh was definitely a buzzword in 2017! The options — linkerd, envoy, istio and conduit — all of which are (or will be) accepted into CNCF sooner or later, provide a glimpse of interesting trend. For instance, [conduit](https://buoyant.io/2017/12/05/introducing-conduit/) is from the same team that built Linkerd. Conduit provides similar capabilities like Linkerd for managing the communication between micro-services, but seems better suited for Kubernetes environment that can run across clouds and with low resource constraints.\n\nLike Service Mesh, the other infrastructure components — logging, monitoring, tracing, and networking are all being containerized (re-engineered) to work well with Kubernetes primitives (resources, pods, policies, federations, labels, taints, tolerations, affinity and anti-affinity, CR, CRDs, Custom Controllers, etc.)\n\n### **_Kubernetes has become a powerful set of nuts and bolts, that is changing the way people should think about infrastructures and how systems are built._**\n\n—\n\nStorage is no different. How data is stored and managed is also being transformed by the possibilities afforded by Kubernetes. Like Service Mesh of initial days, a lot of incumbent storage vendors are providing a patched (which some view as cloud washed or container plugged) solutions that will result in operations and developers spending endless hours firefighting to make them work with cloud native environments.\n\nThe key for any infrastructure component to be called container-native will be characterized by being hardware agnostic and usable at scale! The past few months, there is an active workgroup team grappling with defining — [_Cloud Native Storage (WIP White Paper by CNCF Storage Workgroup)_](https://docs.google.com/document/d/1cJLgOAIWbi-Ya27BY7mjH61yoO3oWcO5tOZYteaDVgI/edit#heading=h.ik4inq9mv6b4)\n\nWhile deliberations are ongoing about what Cloud Native Storage is, which I think will finally be about users adoption, the talks at KubeCon suggest the community sees three distinct storage options for Kubernetes:\n\n- Persistence Volumes from External Storage Providers\n- Local/Ephemeral Storage for Containers\n- Containerized Storage for Containers\n\n—\n\n**Persistence Storage from External Storage Providers**\n\nMost of the cloud providers and incumbent storage vendors want the users to opt for this option where storage is connected via in-tree or out-of-tree dynamic volume provisioners. Many vendors are coming together in helping shape the CSI [(Container Storage Interface)](https://github.com/container-storage-interface/spec), and the initial implementation are slated to get into beta stage in early 2018. There are constant improvements — or strides — being made in storage workflow automation via controllers and `kubectl` — dynamically provisioning volumes, resizing, and snapshots.\n\nI spoke to a number of storage users at KubeCon, including the team at GitHub who are at the forefront of putting Kubernetes in production. The users are still very wary with the state of storage w.r.t using the PVs to connected storage and the amount of work involved in rewriting their operational scripts and playbooks.\n\nAnother issue I heard users talk about that puts them off NAS or SAN — and this was a little surprising as I’ve spent years building a unified storage system that in some environments is really fundamental to the architectures of private clouds and hosting environments — is that they think shared underlying storage does not fit a microservices architecture. Of course, if you read the 12 Factor definition it talks about storage if at all as an attached resource. However — it also is clear from 12 Factor approaches that _dev should be the same as possible as production and that the same people should be doing coding and deploys._ That’s just not the world of external arrays with special teams running storage and different arrays for dev, test, staging, and production.\n\nIt is also worth noting and taking time to understand that these options of connecting to network storage have been around for more than couple of years, and the fact that Stateful workloads on Kubernetes aren’t yet as prevalent says something about user acceptance of the approach! _Users are waiting for better options to be made available — like the support for local storage or something else — but not NAS!_\n\nLearning from the HBO team that was streaming GoT using Kubernetes, it is interesting to see a solution like Rook being used on top of EBS, while EBS is provided as PVs themselves.\n\n![Telemetry](https://cdn-images-1.medium.com/max/800/1*Zl5PPYzJpDZoXrK7DCL_0w.png)\n\n[https://www.youtube.com/watch?v=7skInj_vqN0](https://www.youtube.com/watch?v=7skInj_vqN0)\n\nRook also presented a pretty interesting study against using PVs from external storage to Pods in their talk [here](https://schd.ws/hosted_files/kccncna17/b3/Cloud%20storage-2.pdf). This is inline with what the teams at [PortWorx](https://portworx.com/ebs-stuck-attaching-state-docker-containers/), [StorageOS ](https://schd.ws/hosted_files/kccncna17/ca/2017-12-8-persistent-storage.pdf)and OpenEBS have been advocating as well.\n\n- Make static assignments of disks (physical or virtual) to nodes and use them as local storage — avoid detaching/attaching disks from nodes\n- As long as the applications can take care of replication and sustain longer downtime for nodes and cluster rebuilding times — use local PVs with the storage provisioned in the previous steps.\n- For workloads that don’t inherently support replication, snapshots, etc. use a containerized storage option.\n\n_I am a firm believer in CSI and what it was set to achieve and has already accomplished— Open Standard for interfacing with Storage. Something which SNIA should have done and couldn’t do in past two decades of my experience. OpenSDS seems to be an effort in that direction by SNIA, but is being received with the same cold response from vendors and in turn the community. FWIW, REX-Ray is also playing in the same space._\n\nAt the moment, the focus for CSI is on (simplifying a bit) provisioning and de-provisioning volume, but albeit a good start. But is it enough for the users to start using it? There was an interesting observation made in the F2F storage workgroup meeting at KubeCon that CSI discussions are mostly driven by vendors. Where are the users? Can we say that vendors represent the users, because they interact with their users?\n\nComing from a operational background, for me to consider using CSI to connect with external storage systems, CSI requires to evolve and include API for Day 2+ Operational Usecases that involve — ease of debugging, snapshots, backups, migration and most importantly, a unified monitoring system of the Kubernetes Clusters and the Storage Systems.\n\nDon't get me wrong. We need storage, lots and lots of it and it will be served from external storage systems — cloud (EBS, GPD, etc., ) or on-premise SAN/NAS products. But these external storage systems weren’t designed to be used for micro-services environment but rather to provide volumes to Nodes (physical or virtual) that are long running and are not subject to rapid connects, disconnects and migrations.\n\n_I believe in the long run we will be using CSI with these external storage for what they were designed for — mainly to provision storage to the nodes rather than Pods._\n\n—\n\n**Local/Ephemeral Storage for containers (aka Direct Attached Storage — DAS)**\n\nKubernetes keeps improving the capabilities for managing the local/ephemeral storage. The recent advancements include:\n\n- Support for attaching [block devices](https://schd.ws/hosted_files/kccncna17/8e/Mitsuhiro_Tanino_Block_Volume_KC_CNC_NA17.pdf) to pods\n- Support for enforcing policies or [resource limits for ephemeral storage](https://schd.ws/hosted_files/kccncna17/3e/Kubecon_localstorage.pdf)\n- Enhance the UX for using [local storage for PVs](https://schd.ws/hosted_files/kccncna17/3c/2017%20Kubecon%20Storage%20-%20FINAL.pdf) — dynamic provisioning, hook into the scheduler for pods requiring local storage PVs etc.,\n\n_When using local storage for PVs, the applications using these PVs need to also own up some of the features like — data consistency, replication, snapshots, etc., that are typically taken care of by the storage controllers._\n\n_One of the ongoing issue with using the local storage in clouds are the quirks of disconnecting and connecting the disks to different instances. The local storage is really meant for using storage that is tied to the node — either physically inserted or hardwired to a VM instance._\n\n—\n\n**Containerized Storage for Containers (aka Container Attached Storage — CAS)**\n\nThe appeal for fully containerized storage for containers is in the possibilities that it opens up to the DevOps administrators who are interested in building on-demand programmable infrastructures, which include:\n\n- storage can be observed down to the bit using the same set of tools they use to monitor their compute and network.\n- storage can also be secured using the same tools used to secure application pods\n- storage can be made policy driven similar to networks\n- storage can be programmed and versioned — made an integral part of the work flows for developers and operations administrators\n- storage can also use federation features for cloud migration similar to application pods.\n\n_StorageOS presented at KubeCon on what we call Container Attached Storage — and on how to select which storage approach for which workload and environment. It was a good talk — slides are here: _[talk](https://schd.ws/hosted_files/kccncna17/ca/2017-12-8-persistent-storage.pdf)\n\n_Kubernetes can provide an unified infrastructure layer to the applications by pooling together nodes with compute, network, and **storage as well**._\n\nKubeCon showcased a demo of launching [glusterfs in containers](https://schd.ws/hosted_files/kccncna17/7b/KubeRunningYourStorage1208.pdf). While this is feasible, it might put some hard requirements on the amount of RAM and CPU required for running the software optimized for running in the nodes in containers.\n\nTo be container-native storage, the storage software needs to be broken down into micro-services, just like how Kubernetes runs using micro-services. There has to be greater flexibility provided to the developers and operations to run seamlessly on their choice of hardware!\n\nOpenEBS does just that! OpenEBS provides all the enterprise grade storage features by its open-source containers that can run anywhere. _No kernel dependencies and vendor lock-in._ A typical data path using the OpenEBS Containers is as follows:\n\n![Stateful Apps using OpenEBS Volumes](https://cdn-images-1.medium.com/max/800/1*Ifsa-k-q4EnO7Fpg7E6jLA.png)\n\n[https://github.com/openebs/openebs/blob/master/contribute/design/OpenEBS%20Architecture%20and%20Design.pdf](https://github.com/openebs/openebs/blob/master/contribute/design/OpenEBS%20Architecture%20and%20Design.pdf)\n\nOpenEBS can consume any storage connected to the node and provide enterprise grade storage features (like snapshots, replication, data-consistency, cross-cloud migration, etc.) to Stateful workloads.\n\n2017 saw a steep rise in the community for building OpenEBS with users evaluating it for different types of storage workloads from Cassandra, Minio to MySQL and some users also rolling out services to their customers using Kubernetes and OpenEBS. _I am looking forward to more application work-flow focused automation of Stateful workloads using OpenEBS in 2018._\n\n—\n\nManaging Storage in an enterprise environment — whether it is cloud or on-premise has to be as seamless as interacting with your favorite bot! I know it is a bit far fetched, but it is definitely going to happen in 2018 with companies like MayaData leading from the front!\n\n2017 saw some major improvements to storage in Kubernetes, but there is a lot more to look forward to in 2018!\n\n- CSI spec will mature to encompass all the storage API and will be adopted by a large percentage of storage vendors.\n- Improved debuggability/observability of PV — Metrics and Alerts etc.\n- Make PVs accessible via namespaces and RBAC and extend the Policies to involve HumanOps!\n- Further improvements to resource constraints from the IOPS perspective\n- Support for host-supported file system types to be used on top of local storage\n\n_Programmable and Predictable Infrastructures are what the developers need while the administrators are looking for building infrastructures that can be easily versioned, built, and deployed anywhere — where the economics makes sense._\n\n—\n\nI take tremendous pride in having been associated with MayaData team that is at the forefront of making Storage Operations fade away by extending Kubernetes with containerized storage for containers.\n\nYour participation will shape and accelerate the movement of Stateful Workloads on Kubernetes. Do join us on Slack on either [Kubernetes sig-storage](http://slack.k8s.io/) or [OpenEBS users](http://slack.openebs.io/) or join the [CNCF storage events](https://calendar.google.com/calendar/embed?src=linuxfoundation.org_o5avjlvt2cae9bq7a95emc4740%40group.calendar.google.com)!\n\nLooking forward to an exciting 2018 for the Stateful Workloads on Kubernetes!\n","notHasFeatureImage":true,"slug":"austin-kubecon-—-persistent-storage-round-up-and-looking-beyond!"},{"id":108,"title":"Install OpenEBS using StackPointCloud Trusted Charts?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"07-01-2018","tags":["Digital Ocean","Helm","Kubernetes","Solutions","StackPointCloud"],"excerpt":"What is StackPointCloud Trusted Charts?","content":"\n#### What is StackPointCloud Trusted Charts?\n\n[StackPointCloud](https://stackpoint.io/) (SPC) introduced a concept of Trusted Charts, a list of validated [Helm](https://helm.sh/) Charts provided by its partners to quickly spin up a solution in a [Kubernetes](https://kubernetes.io/) cluster. Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application.\n\nPreviously, I wrote about few different ways of getting OpenEBS up and running on different cloud vendors. Using Helm Chart is one of the available options to deploy OpenEBS. OpenEBS Helm Charts were available since v.5.0 both on [Github](https://github.com/openebs/openebs/tree/master/k8s/charts/openebs) and as a [packaged chart](https://openebs.github.io/charts/). Recently SPC included OpenEBS into their Trusted Charts repo and made it one-click easy for its customers.\n\nSPC Trusted Charts currently offers 23 solutions including databases, CI/CD, monitoring, storage and ingress solutions. Here is the list of Trusted Charts:\n\n### CI/CD\n\n- [Jenkins](https://jenkins-ci.org/)\n- [GitLab Runner](https://docs.gitlab.com/runner/)\n- [Spinnaker](https://www.spinnaker.io/)\n\n### Databases\n\n- [CockroachDB](https://www.cockroachlabs.com/)\n- [Crunchy PostgreSQL Operator](https://github.com/CrunchyData/postgres-operator)\n- [Patroni](https://github.com/turbonomic/kubeturbo)\n- [Redis](https://redis.io)\n- [RethinkDB](https://www.rethinkdb.com/)\n- [MongoDB Replica Set](https://docs.mongodb.com/manual/replication/)\n\n### Ingress/Proxy/Load Balancer\n\n- [Nginx Ingress](https://github.com/kubernetes/ingress-nginx)\n- [Traefik](https://traefik.io/)\n\n### Messaging\n\n- [Kafka](https://kafka.apache.org/)\n- [Rabbitmq](https://www.rabbitmq.com)\n\n### Storage\n\n- [OpenEBS](https://openebs.io/)\n- [Minio](https://www.minio.io/)\n- [Etcd-operator](https://github.com/kubernetes/charts/tree/master/stable/etcd-operator)\n\n### Others\n\n- [Grafana](https://grafana.com/)\n- [Keel](https://keel.sh/)\n- [Kube-lego](https://github.com/jetstack/kube-lego)\n- [Kubeturbo](https://github.com/turbonomic/kubeturbo)\n- [Memcached](https://memcached.org/)\n- [Tensorflow Inception](https://github.com/tensorflow/models/tree/master/research/inception)\n\nI’ll go through the quick steps of deploying OpenEBS.\n\n### Prerequisites\n\nMinimum requirements for deploying your Kubernetes clusters on StackPointCloud:\n\n### Cloud Provider Account\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) or\n- [DigitalOcean](https://www.digitalocean.com)\n\n### Deploy a New Kubernetes Cluster\n\nFirst, go to [stackpoint.io](https://stackpoint.io/) and click on **Launch a Cluster** button to start your free trial.\n\n![Universal control plane for managed Kubernetes](https://cdn-images-1.medium.com/max/800/0*0cB3ttYmslFZgH1h.png)\n\nThen choose your cloud provider. In this example, I will use **Digital Ocean**.\n\n![Choose cloud provider](https://cdn-images-1.medium.com/max/800/0*21G24JgfuqlR6snZ.png)\n\n### Configure Access to Digital Ocean\n\nOn the next screen, we need to configure our provider. You need to provide Digital Ocean API Token and optionally your SSH Key.\n\n![Configure your provider](https://cdn-images-1.medium.com/max/800/0*wDcMg-_HTjIOFvgb.png)\n\nClick on **Add API Token** button.\n\n![Add API token](https://cdn-images-1.medium.com/max/800/0*53wGtQ7eUt18u6pS.png)\n\nAfter you add your credentials, click on **Submit**.\n\n### Configure K8s Cluster\n\nOn “Configure your cluster” page click the edit button on **Distribution** and choose **Ubuntu 16.04 LTS**.\n\n![Configure your cluster](https://cdn-images-1.medium.com/max/800/0*NvtnryAA8GNi-fyN.png)\n\nChange the **Cluster Name** something meaningful like **OpenEBS Demo**.\n\n![Enter cluster name](https://cdn-images-1.medium.com/max/800/0*LTa6zBooJdTsyqss.png)\n\nLeave everything else as default and click on **Submit**.\n\nIn about 10–15 minutes you will get your new cluster deployed.\n\n### Adding OpenEBS to Your Kubernetes Cluster\n\nFirst, make sure your cluster and all nodes are up.\n\nOn the **Control Plane** tab click on your recently created cluster.\n\n![Control plane](https://cdn-images-1.medium.com/max/800/0*RHQ9LbyxydjHkJSk.png)\n\nOnce the Kubernetes cluster is up on Digital Ocean with functional Helm, scroll down to the **Solutions** tab and click on **Add Solution** button.\n\n![Add Solution](https://cdn-images-1.medium.com/max/800/0*sH0lzv23vHonV5Zk.png)\n\nClick on **Add Solutions**, and select **Trusted Charts**.\n\n![Select charts](https://cdn-images-1.medium.com/max/800/0*V6iP5PzNAzFk4sME.png)\n\nFrom the list above select **OpenEBS**.\n\n![OpenEBS namespace](https://cdn-images-1.medium.com/max/800/0*CJkPrkJCS9Fp_GXu.png)\n\n**Release Name** is randomly generated every time. If you want to use OpenEBS example workloads provided in OpenEBS repos without any modification then use `default` as **NameSpace**. Otherwise, you need to modify the namespace for workloads you deploy and make sure to use the same name.\n\nClick on **Install** to deploy OpenEBS on your cluster.\n\n**Note:** Default settings assume that RBAC is enabled. If you disabled RBAC while you are configuring your provider previously then set `rbacEnable: false` otherwise use default values.\n\nState field should be green after OpenEBS is successfully added.\n\n![OpenEBS successfully added](https://cdn-images-1.medium.com/max/800/0*HzCZp3Z5LbT3Hsrh.png)\n\nNow your cluster is ready; you can run your workloads on `openebs-standard` and other predefined storage classes.\n\nTo confirm, click on **Kubernetes Dashboard**. This will bring up your Kubernetes Dashboard UI in a new window. You will find all predefined OpenEBS **Storage Classes** here under **Storage Classes** section.\n\n![Kubernetes Storage Classes](https://cdn-images-1.medium.com/max/800/0*mNU-nhwvNy9UB0W5.png)\n\nNow you are ready to deploy your stateful workloads.\n\nTake a look at my previous articles on step-by-step instructions for deploying few popular stateful workloads such as [Cassandra](http://containerized.me/how-to-deploy-a-cassandra-cluster-ring-on-kubernetes-openebs/), [Jenkins](http://containerized.me/how-to-deploy-jenkins-on-kubernetes-openebs/), and [Postgres](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/) on OpenEBS persistent storage.\n\n---\n\n_Originally published at _[_Containerized Me_](http://containerized.me/install-openebs-using-stackpointcloud-trusted-charts/)_._\n","notHasFeatureImage":false,"slug":"install-openebs-using-stackpointcloud-trusted-charts?"},{"id":109,"title":"OpenEBS 0.5 enables Storage Policies for Kubernetes Persistent Volumes","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"30-11-2017","tags":["Container","Open Source","Storage Containers","Kubernetes","Updates"],"excerpt":"Personally, it is very exciting and enriching to see the growth of the OpenEBS project — from its capabilities, contributors and community perspective!","content":"\n\nPersonally, it is very exciting and enriching to see the growth of the OpenEBS project — from its capabilities, contributors and community perspective!\n\nI believe the metric to measure the success of OpenSource Projects is determined by the number of users and the conversion ratio of users turning into contributors. The past couple of months (thanks to campaigns like Hacktoberfest and the ongoing OpenEBS Hackfest), we have seen a steep rise in the number of contributors and contributions to the OpenEBS project.\n\n![OpenEBS contributors](https://cdn-images-1.medium.com/max/800/1*BMOr9ULh_7KnM6k8aUj9hw.png)\n\nIn almost all the interactions we had with the user community, we are seeing a clear resonance of the value proposition that OpenEBS brings to the [DevOps teams managing systems with large number of micro-services](https://twitter.com/muratkarslioglu/status/921072858628997121). The best part is that the users are independently evaluating OpenEBS and finding ways to automate their Compute, Network, Storage, and Data related Operations.\n\n![Towards Kubernetes](https://cdn-images-1.medium.com/max/800/0*XilwHl_ucs5K5fcK.jpg)\n\nToday, I can look back on our decision to use Kubernetes as a framework to build OpenEBS is one of the best decisions we have made. While the core of the Kubernetes community is helping DevOps teams treat Compute and Network as Code — we at OpenEBS are focused at extending Kubernetes to enable treating Storage and Data also as Code.\n\nI am delighted to announce that OpenEBS 0.5 is released with 300+ PRs coming from 50+ new community contributors, with several new features and bug fixes. Summary of changes are available in the [Release Notes](https://github.com/openebs/openebs/releases/tag/v0.5.0).\n\nSome notable changes include:\n\n- Storage Policy Enforcement Framework that allows DevOps teams to deploy a customized storage\n- Extend OpenEBS API Server to expose volume snapshot API\n- Support for deploying OpenEBS via helm charts\n- Support for monitor and get insights into OpenEBS volumes via Prometheus and Grafana\n- Sample Deployment YAMLs and corresponding Policy enabled Storage Classes for several stateful applications\n- Sample Deployment YAMLs for launching Kubernetes Dashboard for a preview of the changes done by OpenEBS Team to Kubernetes Dashboard\n\n***My favorite capability is the Storage Policy Framework that will enable each DevOps team to have their own storage controller — with their own storage policies. And the possibilities it will open-up!***\n\nImagine as a Developer I want to test my service against MySQL database with different datasets. On my staging PV (mysqldata), I can create multiple snapshots containing different datasets — say snaps like *sandy* and *wendy*. Now I can extend OpenEBS to support a new policy *\"openebs.io/jiva-replica-snap\"*, that can launch a new volume using snapshot for seed data.\n\nThe policy can be defined for snap — *sandy* as follows:\n\n # Define a SC referring to snapshot sandy\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mysvc-mysqldata-kiran-ds-sandy\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/jiva-replica-count: \"1\"\n openebs.io/jiva-source-pv: \"mysqldata\" \n openebs.io/jiva-replica-snap: \"mysqldata-ds-sandy\"\n\nand another one for *wendy*:\n\n # Define a storage classes supported by OpenEBS\n apiVersion: storage.k8s.io/v1\n kind: StorageClass\n metadata:\n name: mysvc-mysqldata-kiran-ds-wendy\n provisioner: openebs.io/provisioner-iscsi\n parameters:\n openebs.io/jiva-replica-count: \"1\"\n openebs.io/jiva-source-pv: \"mysqldata\"\n openebs.io/jiva-replica-snap: \"mysqldata-ds-wendy\"\n\nIn my PVC, I can now use the above StorageClasses (augmented with OpenEBS Storage Policies) to point to the different datasets and independently test my service.\n\nAll this from ***kubectl***.\n\nIn the background, OpenEBS will create a separate Containerized Storage Engine/Controller for my test database — by sourcing the data from the specified snapshot.\n\nAnd btw, we are on the [CNCF LandScape](https://github.com/cncf/landscape) under the Cloud Native Storage options, and decidedly leading the niche market for Containerized Storage Controller. You will hear more on this in the coming days at the KubeCon, Austin.\n\n![CNCF](https://cdn-images-1.medium.com/max/800/1*rdKFLGyf0hRDB_zcgGlywA.png)\n\nYou will hear more on OpenEBS 0.5 in the coming days at `KubeCon 2017`, Austin, Texas.\n\nI will be at the OpenEBS booths with my team and look forward to catching up with some of you in person.\n","notHasFeatureImage":true,"slug":"openebs-0.5-enables-storage-policies-for-kubernetes-persistent-volumes"},{"id":110,"title":"Storage Policies — It’s different this time","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"29-11-2017","tags":["Container","DevOps","Kubernetes","Storage Policy","Updates"],"excerpt":"One of the most common disbelief at the operator’s end would be the reports of an application’s (that consumed this storage) sudden death after introducing a much awaited shiny new storage feature.","content":"\n\n### Need for Storage Policy\n\nOne of the most common disbelief at the operator’s end would be the reports of an application’s (*that consumed this storage*) sudden death after introducing a much awaited shiny new storage feature. To make things worse the same surprise would be reciprocated from the storage provider as well. Neither the operator nor the storage fellow have any clues to this sudden strangeness.\n\nHaving gone through many such cycles of frustrations building storage features that satisfy the needs of every operator as well does not break stuff that are outside its control; I knew there was clearly a gap in storage design that needs to be talked about openly and bridged. In this article, I present storage policy as the solution to this pressing problem.\n\nThe moment I think about conceptualizing storage policy, my mind tries to battle against many odds. *Is this not what everyone (*read storage vendors*) has accomplished?*\n\n**Think again!!!** The chances are, it works for specific storage **“A”** and needs a radically different approach for the other storage **“B”**; assuming both of these provide the infrastructure for the operator’s storage needs. So the operator goes ahead and does some plumbings if lucky enough to get these infrastructures as white-box components. *At the end of the day, the operator is ready with a new bespoke program and feels so lucky*. Whatsoever, this does not last long due to the very behaviour of infra components. *This operator needs to be ready all the time for the next upgrade to above crafted program, **since little can be expected from the monolith storage fellas*. *Poor operator is left to fend for self, building all possible defence mechanisms*.\n\nGetting back to the need for a storage policy; *what is the edge in this incarnation of storage policy then*? Well, it is all about capitalising the essential features that being native to cloud and being operated from within a container provides *(I can only hope and pray for those storage controllers who cannot operate from the cloud(s) or can not be run from within the container(s))*.\n\nHaving said that, there is no innate need to design storage policy that is drastically different just to prove its greatness. Instead, there is a need to design, build, and apply policies that are agnostic to a storage implementation and is conducive to storage operations that are never ever meant to be interrupted.\n\n1. A Storage Policy should be simple to setup on Day 0\n2. It should be simple to expect the obvious on Day 1\n3. It should be simple enough to build an update whenever desired.\n\n### Is it that simple?\n\nAll this time, I have been describing the storage policy and its operations as simple. *Well, they can be described as simple, but not easy*. What was your feeling when you created one last time? *Have you ever felt creating a storage snapshot policy is easy?* How about a policy that does periodic restore\nof some randomly picked snapshots and marks them as *PASSED* or *FAILED* after verifying its data integrity. How did that failure alert you? Was it a slack notification? To make it further interesting, can you think of a snapshot policy that hooks into its consuming application’s life cycle events before trying out the snapshot (*Yes!!! I am suggesting those ‘freeze’ and ‘thaw’ exploits before and after a snapshot*).\n\n**Policies can be described as simple, but not easy.**\n\nRemember, these are not the only policies that can be thought of with respect to storage (*we have not even scratched the storage surface*). While all of these are achievable in a cloud and container native way, they may not be termed as easy.\n\nApplying storage policies is like a dangerous yet interesting sport. Like rock climbing, building a suitable policy can be learnt and then adapted based on the needs, workloads, environment, and so on. However, these policies if applied incorrectly can create a ripple-effect that can lead to increased costs and SLA misses which in turn leads to more support personnel on duty, and so on. As we read further, this new design of storage policies helps in eliminating the aforementioned impacts.\n\n### Bring your own YAML\n\nDid I just say YAML? Yes, I did and I shall explain it in a moment. Let us first explore the possible design angles of a storage policy. *Will it not be great to design a policy that fits well within an enterprise’s existing processes, its employees, its culture, its tools?* Is this too much to ask for?\n\nNo!!! *These are no more a set of good to have features but a must have checklist.* I believe expressing storage policy intents as YAMLs can pass all possible checks in an enterprise’s checklist. These current generation declarative intents are now fluent enough to be understood by APIs. In addition, one of YAML’s greatest strengths is its ability to abstract the entire logic that most of us understand as programmable code. This code now becomes truly yours since you have full control of these YAMLs. In other words, it is the operators who have the controlling rights.\n\nTo reiterate, there are few but really solid facts that makes this approach towards policy design a much better one than that of all its predecessors.\n\n***Here are my bets that makes this design different:***\n\n***Fact #1*** — To put things into perspective, the unassuming reader needs to look at the storage policies along with the current trends in cloud infrastructure as well as the trends in container engine. The cloud has of late become seamless. Thanks to Kubernetes which has been continuously bridging the impedance mismatch between different cloud providers. In addition, the communities involved in Container Storage Initiative, Container Runtime Interface, etc. are making intents as first class citizens. These declarative code pieces are thought of in a bottoms-up approach in each of these implementations which are then placed together coherently by the likes of Kubernetes. Now this is what is definitely more effective than just a tool that parses YAML and runs in isolation.\n\n***Fact #2*** — These intents are precise and parsed with appropriate validation checks to state with decisive control on the exact outcome. In other words they control the actual execution logic in addition to accepting input values via its declarative specifications. This grounds-up approach coupled with the nativity towards cloud as well as container engine is more suited to design storage policies that align with modern day DevOps’ practices.\n\n***Fact #3*** — Let us not forget the containers and the critical role they play in this age of cloud and orchestrators. Perhaps containers have become so ubiquitous and hence are easy to miss. I truly feel there are umpteen number of cool things that are yet to be discovered when we run a storage controller inside a container.\n\n**Fact #4** — All these also mean the intents that were once the sole prerogative of humans can now be built and operated by machines as well. This too with the same ease that the humans used to enjoy. After all, the ingredients *(#1, #2 and #3)* to make this possibility are all in place. This has come of age and is really an advancement in my opinion.\n\n***Bringing your own YAML really means setting your own policies and having complete control over their execution as well.***\n\n### Use the tool(s) you always loved\n\nA policy alone cannot justify its existence unless it is complimented with simple tools and automated processes. The careful reader might have already guessed it. Yes, I am talking of aligning storage policies with DevOps to realize its full potential.\n\n*Making storage policies more visible, more obvious, and enabling them to the enterprise’s established processes will make them simple to be believed and instil the faith to operate*. *Once again, there is a learning involved but the curve is not steep*.\n\nThese policies should offer the finest levels of control to the operators’ tools, their bots, and of course operators themselves when such a need arises. *For example finer granularity is craved for during rollbacks, automated downgrades, or blue-green deployments which are not uncommon in the world of storage infrastructures.*\n\nTo complete the DevOps cycle, these policies which can be handcrafted or system-generated can be submitted to the approver(s) (*which again can be a combination of humans as well as their loved tools*) as Pull Requests before being installed and applied against the storage.\n\n### Storage was the proverbial “Missing Cog”\n\nWe are seeing users finally achieving what many have dreamt of for so long  —  ***storage****(and the rest of the infrastructure)* truly being driven by the needs of the application and in a way that remains understandable and for a variety of reasons *(take infra as code for example)* trusted by humans. And now increasingly we see the recognition that containerised storage itself is another important ingredient.\n\nThanks to [OpenEBS](http://openebs.io) and more broadly containerised storage. *For the first time, every team and workload can have its **own fully functional storage controller**, with capabilities that have always been required by enterprise storage systems and that are still useful in taking care of *stateful workloads*. Our users do not want to give up the tools they used for the care and feeding of MySQL for example just because it now runs in a container. *This is possible by enabling capabilities like snapshots, versioning, encryption, and more **as knobs** to be able to be turned on/off for each workload.*\n\nThis incarnation of storage policies make it easy for many procedures for these workloads to be recorded as YAML; *the run book is truly code and so can easily be shared, versioned, and executed without humans having to play a role in the ugly details of managing storage system **A** or **B** or even **C**. Storage fades into the background. Time is ripe for the operators to rule.*\n","notHasFeatureImage":true,"slug":"storage-policies-—-it’s-different-this-time"},{"id":111,"title":"How to deploy a Cassandra Cluster/Ring on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"25-11-2017","tags":["Cassandra","Kubernetes","NoSQL","Solutions"],"excerpt":"Apache Cassandra is a distributed key-value store intended to run in a data center and also across multiple data centers. Initially it was designed as Facebook as an infrastructure for their messaging platform.","content":"\n\nApache Cassandra is a **distributed key-value store** intended to run in a data center and also across multiple data centers. Initially, it was designed as Facebook as an infrastructure for their messaging platform. Later it is open sourced, and today it’s one of the most active Apache projects.\nIf you are using eBay, Twitter, Spotify, or Netflix you are consuming data provided by Cassandra. For example, Netflix uses Cassandra to keep track of your current place in a streaming video, as well as movie ratings, bookmarks, and user history of 90+ million users.\nAmazing to see how much of this technology we consume in our day-to-day life. The feature that allowed me and my wife to start watching Stranger Things on our long trip on a tablet and continued on TV was depending on Cassandra. To give you an idea of its size, according to a [recent presentation](https://www.youtube.com/watch?v=2l0_onmQsPI), Cassandra serving Netflix has 250+ Clusters, 10,000+ Nodes, and 3+ PB of data.\nIn summary, Cassandra solves the problem of mapping the key-value pair to a server/node, in our case to a container. This mapping is called the **partitioner**. There are two common placement strategies used by Cassandra: **SimpleStrategy** or **NetworkTopologyStrategy**. SimpleStrategy uses partitioner Murmur3Partitioner by default. Both **Murmur3Partitioner** and **RandomPartitioner** partitioners uniformly distribute data to nodes across the cluster. Read and write requests to the cluster are evenly distributed while using these partitioners. Load balancing is simplified as each part of the hash range receives an equal number of rows on average. **Byte-Order Partitioner **is not recommended other than key range queries.\n\nFor development work, the SimpleStrategy class is acceptable. For production work, the NetworkTopologyStrategy class must be set. In production, you will end up with multiple rings using mostly NetworkTopology placement which is by itself extremely complex to plan.\nIf you want to learn the architecture of Cassandra, the University of Illinois has a great course on [Cloud Computing Concepts](https://www.coursera.org/learn/cloud-computing/home/welcome) and [Key-Value Stores](https://www.coursera.org/learn/cloud-computing/home/week/4) which covers internals of Cassandra. You can also find more about custom SeedProvider and Snitches [here](https://github.com/kubernetes/kubernetes/issues/24286).\nCassandra doesn’t like shared storage, therefore use of NFS or GlusterFS not recommended for Cassandra rings. It’s also recommended to use SSD or NVMe, since it’s essential to have low latency random reads and good sequential writes at the same time. These kinds of requirements can be only satisfied with OpenEBS like local and persistent storage solutions.\nTo achieve the best fault tolerance with Cassandra, you need to have an excellent understanding of the [**snitch**](http://cassandra.apache.org/doc/latest/operating/snitch.html)and placement strategies. There is a big debate on whether if Cassandra or the storage should handle the placement of data. My suggestion would be to have a balanced approach and have both. OpenEBS can help you to place your persistent volumes across the datacenter, multiple cloud vendors, and fault domains against Cassandra replica failures. First, you can **avoid rebalancing** your cluster in case of a datacenter failure. Second, in case of node failures in a rack, you can bring up the same node from a snapshot and **minimize the time needed to rebalance**.\n\nI will use Cassandra custom Kubernetes SeedProvider that allows discovery within Kubernetes clusters as they join the cluster and deploy using `gcr.io/google-samples/cassandra:v11` image from Google’s container registry.\n\nLet’s deploy our first three replica Cassandra cluster on our existing AWS K8s cluster with OpenEBS. If you are using on local minikube or datacenter, you can keep the default **SimpleStrategy** and **Murmur3Partitioner** in `cassandra.yaml` file.\n\n#### **Prerequisites**\n\n**Software**\n\n- [Docker ](https://docs.docker.com/engine/installation/)installed\n- Kubernetes 1.8.3+ RBAC enabled cluster\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n- Cassandra 3.x\n\n**Cloud Provider**\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n#### **Deploy Cassandra Stateful with Persistent Storage in one Region**\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following steps to launch a Cassandra service with any number of nodes you like.\n\nBy using environment variables, you can change values that are inserted into `cassandra.yaml`. Default **endpoint_snitch** is set to SimpleSnitch. I will change the Snitch to Ec2Snitch and also increase the replicas from 3 to 4 later.\n\nBefore getting started, check the status of the cluster using the following command.\n\n kubectl get nodes\n\nOn my setup, I have one master and four worker nodes on AWS in the same US West (Oregon) region and availability zone (us-west-2a).\n\n ubuntu@ip-172–23–1–236:~$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n ip-172–23–1–225.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–236.us-west-2.compute.internal Ready master 21h v1.8.3\n ip-172–23–1–32.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–35.us-west-2.compute.internal Ready 21h v1.8.3\n ip-172–23–1–46.us-west-2.compute.internal Ready 21h v1.8.3\n\nDownload OpenEBS GitHub repo to your host, where sample YAML files are stored.\n\n git clone [https://github.com/openebs/openebs.git](https://github.com/openebs/openebs.git)\n\nFirst list predefined OpenEBS storage classes available to you.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get storageclasses\n NAME PROVISIONER\n default kubernetes.io/aws-ebs\n etcd-backup-gce-pd kubernetes.io/gce-pd\n gp2 (default) kubernetes.io/aws-ebs\n openebs-cassandra openebs.io/provisioner-iscsi\n openebs-es-data-sc openebs.io/provisioner-iscsi\n openebs-jupyter openebs.io/provisioner-iscsi\n openebs-kafka openebs.io/provisioner-iscsi\n openebs-mongodb openebs.io/provisioner-iscsi\n openebs-percona openebs.io/provisioner-iscsi\n openebs-redis openebs.io/provisioner-iscsi\n openebs-standalone openebs.io/provisioner-iscsi\n openebs-standard openebs.io/provisioner-iscsi\n openebs-zk openebs.io/provisioner-iscsi\n\nGo to `openebs/k8s/demo/cassandra/` folder and edit `cassandra-statefulset.yaml` file.\n\n vi cassandra-statefulset.yaml\n\nThis file should look like below. You can edit and specify number of replicas preferred and your own OpenEBS storage class before applying.\n\n apiVersion: apps/v1beta1\n kind: StatefulSet\n metadata:\n name: cassandra\n labels:\n app: cassandra\n spec:\n serviceName: cassandra\n replicas: 3\n selector:\n matchLabels:\n app: cassandra\n template:\n metadata:\n labels:\n app: cassandra\n spec:\n containers:\n — name: cassandra\n image: gcr.io/google-samples/cassandra:v11\n imagePullPolicy: Always\n ports:\n — containerPort: 7000\n name: intra-node\n — containerPort: 7001\n name: tls-intra-node\n — containerPort: 7199\n name: jmx\n — containerPort: 9042\n name: cql\n resources:\n limits:\n cpu: “500m”\n memory: 1Gi\n requests:\n cpu: “500m”\n memory: 1Gi\n securityContext:\n capabilities:\n add:\n — IPC_LOCK\n lifecycle:\n preStop:\n exec:\n command: [“/bin/sh”, “-c”, “PID=$(pidof java) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done”]\n env:\n — name: MAX_HEAP_SIZE\n value: 512M\n — name: HEAP_NEWSIZE\n value: 100M\n — name: CASSANDRA_SEEDS\n value: “cassandra-0.cassandra.default.svc.cluster.local”\n — name: CASSANDRA_CLUSTER_NAME\n value: “K8Demo”\n — name: CASSANDRA_DC\n value: “DC1-K8Demo”\n — name: CASSANDRA_RACK\n value: “Rack1-K8Demo”\n — name: CASSANDRA_AUTO_BOOTSTRAP\n value: “false”\n — name: POD_IP\n valueFrom:\n fieldRef:\n fieldPath: status.podIP\n readinessProbe:\n exec:\n command:\n — /bin/bash\n — -c\n — /ready-probe.sh\n initialDelaySeconds: 15\n timeoutSeconds: 5\n # These volume mounts are persistent. They are like inline claims,\n # but not exactly because the names need to match exactly one of\n # the stateful pod volumes.\n volumeMounts:\n — name: cassandra-data\n mountPath: /cassandra_data\n volumeClaimTemplates:\n — metadata:\n name: cassandra-data\n annotations:\n volume.beta.kubernetes.io/storage-class: openebs-cassandra\n spec:\n accessModes: [ “ReadWriteOnce” ]\n resources:\n requests:\n storage: 5G\n\nNote: There are few parameters you may want to modify.\n\n`apiVersion: apps/v1beta2` API group and version is introduced 1.8 release.\n\n`replicas: 3`, I’m starting with 3 replicas and will increase later.\n\n`image: gcr.io/google-samples/cassandra:v12` is the latest image available at the time I've tested.\n\n`volume.beta.kubernetes.io/storage-class: openebs-cassandra` I’m using a predefined OpenEBS storage class. You can modify it separately.\n\n#### Create a Cassandra Headless Service\n\nTo be able to have a simple discovery of the Cassandra seed node we need to create a “headless” service. If you view the `cassandra-service.yaml`file, you will notice that `clusterIP` is set to None. This will allow us to use KubeDNS for the Pods to discover the IP address of the Cassandra seed.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cat cassandra-service.yaml\n apiVersion: v1\n kind: Service\n metadata:\n labels:\n app: cassandra\n name: cassandra\n spec:\n clusterIP: None\n ports:\n — port: 9042\n selector:\n app: cassandra\n\nNow apply `cassandra-service.yaml` file to create headless service.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl apply -f cassandra-service.yaml\n service “cassandra” created\n\n#### Create a Cassandra StatefulSet\n\nMost applications deployed on Kubernetes should be **cloud-native** and rely on external resources for their data and state. However, stateful application and databases like Cassandra require stateful sets and persistent volumes to ensure resiliency. In this case, OpenEBS will provide us our persistent volume.\n\nThe StatefulSet is responsible for creating the Pods. Run the following command to start our Cassandra replicas.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl apply -f cassandra-statefulset.yaml\n statefulset “cassandra” created\n\n#### Validate the StatefulSet\n\nCheck if your StatefulSet has deployed using the command below. Time may take around 4–5 minutes to complete.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 3 3 5m\n\nIf you don’t see all 3 replicas ready, you can check status of pods to watch progress. For example, I ran `kubectl get pods` after 2 minutes below. First node was ready and second was still creating. All three pods were ready after around 5 minutes.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n cassandra-0 1/1 Running 0 2m\n cassandra-1 0/1 ContainerCreating 0 12s\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 13m\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 13m\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 12s\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 0/1 ContainerCreating 0 12s\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 12s\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 2m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 2m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0\n\nVerify that all the OpenEBS persistent volumes are created, the Cassandra headless service and replicas are running.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods -o wide\n NAME READY STATUS RESTARTS AGE IP NODE\n cassandra-0 1/1 Running 0 6m 10.2.2.4 ip-172–23–1–32.us-west-2.compute.internal\n cassandra-1 1/1 Running 0 4m 10.2.4.6 ip-172–23–1–46.us-west-2.compute.internal\n cassandra-2 1/1 Running 0 2m 10.2.1.6 ip-172–23–1–35.us-west-2.compute.internal\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 17m 10.2.4.3 ip-172–23–1–46.us-west-2.compute.internal\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 17m 10.2.3.2 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 4m 10.2.3.4 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 1/1 Running 0 4m 10.2.4.5 ip-172–23–1–46.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 4m 10.2.3.5 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-9c4bfcd6–4ss2r 1/1 Running 0 2m 10.2.1.4 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45–7tss8 1/1 Running 0 2m 10.2.3.6 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45-vfkrn 1/1 Running 0 2m 10.2.1.5 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 6m 10.2.1.3 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 6m 10.2.3.3 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0 6m 10.2.4.4 ip-172–23–1–46.us-west-2.compute.internal\n\nOn the list of the Pods above, you see 3 Pods running. Your Pod names should be cassandra-0, cassandra-1, cassandra-2 and the next pods would follow the ordinal number (cassandra-3, cassandra-4,...). Use this command to view the Pods created by the StatefulSet:\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get svc\n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n cassandra ClusterIP None 9042/TCP 7m\n kubernetes ClusterIP 10.3.0.1 443/TCP 22h\n maya-apiserver-service ClusterIP 10.3.0.204 5656/TCP 18m\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.188 3260/TCP,9501/TCP 5m\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.23 3260/TCP,9501/TCP 3m\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-svc ClusterIP 10.3.0.187 3260/TCP,9501/TCP 7m\n\n#### Verifying Successful Cassandra Deployment\n\nCheck if the Cassandra nodes are up, perform a **`nodetool status`** on cassandra-0 node :\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl exec -ti cassandra-0 — nodetool status\n Datacenter: DC1-K8Demo\n ======================\n Status=Up/Down\n |/ State=Normal/Leaving/Joining/Moving\n — Address Load Tokens Owns (effective) Host ID Rack\n UN 10.2.4.6 83.17 KiB 32 75.6% 3c93c7b7–61a7–4cf1-a407-cb47b1de0763 Rack1-K8Demo\n UN 10.2.1.6 65.65 KiB 32 59.5% 552569fe-c6df-4edb-a553–9efdcf682fb3 Rack1-K8Demo\n UN 10.2.2.4 83.12 KiB 32 64.9% 92060271-d8cd-48be-a489-c830a8553462 Rack1-K8Demo\n\nUN means node is **up** and in **normal** state. You will also notice that each node has 32 tokens. This is the default value, in production workloads, a good default value for this is 256. See more information [here](http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/architecture/architectureDataDistributeVnodesUsing_c.html).\n\nThe **Owns** column suggests the data distribution percentage for the content placed into the Cassandra keyspaces.\n\n#### Create a Test Keyspace with Tables\n\nIdentify the IP Address of any of the Cassandra replicas, for example, Cassandra-0. This is available from the output of the `nodetool status` command executed above (10.2.4.6, 10.2.1.6, 10.2.2.4).\n\n**Cqlsh** is a Python-based utility that enables you to execute Cassandra Query Language (CQL). **CQL** is a declarative language that allows users to query Cassandra using semantics similar to SQL.\n\nInstall the python-minimal and python-pip apt packages and perform a pip install of Csqlsh using the following commands.\n\n sudo apt-get install -y python-minimal python-pip\n pip install cqlsh\n\nLogin to the CQL shell using the Cqlsh utility using the following command.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cqlsh 10.2.4.6 9042 — cqlversion=”3.4.2\"\n Connected to K8Demo at 10.2.4.6:9042.\n [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]\n Use HELP for help.\n cqlsh>\n\nAs I mentioned earlier, you have two placement options while creating a keyspace. You can either use SimpleStrategy or NetworkTopologyStrategy.\n\nYou can create a keyspace using SimpleStrategy with replication factor 2 by running the following commands.\n\n cqlsh> create keyspace ssks with replication = { ‘class’ : ‘SimpleStrategy’ , ‘replication_factor’ : 2 };\n\nI will create a keyspace using NetworkTopologyStrategy by running the following commands.\n\n cqlsh> create keyspace ntsks with replication = { ‘class’ : NetworkTopologyStrategy’, ‘DC1-K8Demo’ : 1 };\n\n cqlsh> describe keyspaces;\n\n ntsks system_schema system_auth system system_distributed system_traces\n\nTo use NetworkTopologyStrategy with data centers in a production environment, you need to change the default snitch, **SimpleSnitch** to a network-aware **Ec2Snitch**. You need to define one or more data center names in the snitch properties file, and use those data center names to define the keyspace; otherwise, Cassandra will fail to find a node. You can find the instructions to change the default Snitch [here](https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_switch_snitch.html).\n\nCreate a table with test content and view the data using the following commands.\n\n cqlsh> use ntsks;\n\n cqlsh:ntsks> create table inventory (id uuid,Username text,Email text,Age int,PRIMARY KEY ((id), Username));\n\n cqlsh:ntsks> insert into inventory (id, Username, Email, Age) values (1234b130-ae79–11e4-ab27–0700500c9a24, ‘Murat’, ‘murat@cloudbyte.com’, 40);\n\n cqlsh:ntsks> select * from inventory;\n\n id | username | age | email\n — — — — — — — — — — — — — — — — — — — + — — — — — + — — -+ — — — — — — — — — — -\n 1234b130-ae79–11e4-ab27–0700500c9a24 | Murat | 37 | murat@cloudbyte.com\n\n (1 rows)\n\nFlush the data to ensure it is written to a disk from the memtable (memory) using the following command.\n\n kubectl exec cassandra-0 — nodetool flush ntsks\n\n#### Delete the Test Keyspace\n\nVerify the masterless nature of Cassandra StatefulSet by deleting the keyspace from another replica, in this example, Cassandra-1.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ cqlsh 10.2.1.6 9042 — cqlversion=”3.4.2\"\n Connected to K8Demo at 10.2.1.6:9042.\n [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]\n Use HELP for help.\n cqlsh> use ntsks;\n\n cqlsh:ssks> select * from Inventory;\n\n id | username | age | email\n — — — — — — — — — — — — — — — — — — — + — — — — — + — — -+ — — — — — — — — — — -\n 1234b130-ae79–11e4-ab27–0700500c9a24 | Murat | 37 | murat@cloudbyte.com\n\n (1 rows)\n\n cqlsh> drop keyspace ntsks;\n\nVerify that the keyspace is deleted successfully using the following command.\n\n cqlsh> describe keyspaces\n\n system_traces system_schema system_auth system system_distributed\n\n#### Scale the StatefulSet\n\nTo increase or decrease the size of your StatefulSet you can use the scale command:\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl scale — replicas=4 statefulset/cassandra\n statefulset “cassandra” scaled\n\nWait a minute or two and check if it worked:\n\n $ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 4 4 1h\n\nIf you watch the Cassandra pods deploy, they should be created sequentially.\n\nYou can view the list of the Pods again to confirm that your Pods are up and running.\n\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get statefulsets\n NAME DESIRED CURRENT AGE\n cassandra 4 4 1h\n ubuntu@ip-172–23–1–236:~/openebs/k8s/demo/cassandra$ kubectl get pods -o wide\n NAME READY STATUS RESTARTS AGE IP NODE\n cassandra-0 1/1 Running 0 1h 10.2.2.4 ip-172–23–1–32.us-west-2.compute.internal\n cassandra-1 1/1 Running 0 1h 10.2.4.6 ip-172–23–1–46.us-west-2.compute.internal\n cassandra-2 1/1 Running 0 1h 10.2.1.6 ip-172–23–1–35.us-west-2.compute.internal\n cassandra-3 0/1 Running 0 1m 10.2.3.9 ip-172–23–1–225.us-west-2.compute.internal\n maya-apiserver-5994b58bbb-ss7mr 1/1 Running 0 1h 10.2.4.3 ip-172–23–1–46.us-west-2.compute.internal\n openebs-provisioner-6f45dcf459-hqldl 1/1 Running 0 1h 10.2.3.2 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-ctrl-9c7dcdcfc-bgmrp 1/1 Running 0 1h 10.2.3.4 ip-172–23–1–225.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–572j8 1/1 Running 0 1h 10.2.4.5 ip-172–23–1–46.us-west-2.compute.internal\n pvc-13a2ebce-d226–11e7–955b-062af127ae24-rep-78bf89ff99–66qtf 1/1 Running 0 1h 10.2.3.5 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-ctrl-9c4bfcd6–4ss2r 1/1 Running 0 1h 10.2.1.4 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45–7tss8 1/1 Running 0 1h 10.2.3.6 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5383da78-d226–11e7–955b-062af127ae24-rep-5bbbd9ff45-vfkrn 1/1 Running 0 1h 10.2.1.5 ip-172–23–1–35.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-ctrl-5b6d99869–7gxv5 1/1 Running 0 1m 10.2.3.7 ip-172–23–1–225.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-rep-5fc8b95cd-6vfbt 1/1 Running 0 1m 10.2.2.5 ip-172–23–1–32.us-west-2.compute.internal\n pvc-5c9e5136-d22f-11e7–955b-062af127ae24-rep-5fc8b95cd-h22qz 1/1 Running 0 1m 10.2.3.8 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-ctrl-584956b667-n88mv 1/1 Running 0 1h 10.2.1.3 ip-172–23–1–35.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84–5m8nz 1/1 Running 0 1h 10.2.3.3 ip-172–23–1–225.us-west-2.compute.internal\n pvc-d13bf437-d225–11e7–955b-062af127ae24-rep-74d4cf4b84-c4t9c 1/1 Running 0 1h 10.2.4.4 ip-172–23–1–46.us-west-2.compute.internal\n\nYou can perform a `nodetool status` to check if the other Cassandra nodes have joined and formed a Cassandra cluster.\n\n $ kubectl exec -ti cassandra-0 — nodetool status\n Datacenter: DC1-K8Demo\n ======================\n Status=Up/Down\n |/ State=Normal/Leaving/Joining/Moving\n — Address Load Tokens Owns (effective) Host ID Rack\n UN 10.2.4.6 174.97 KiB 32 59.7% 3c93c7b7–61a7–4cf1-a407-cb47b1de0763 Rack1-K8Demo\n UN 10.2.1.6 182.32 KiB 32 43.8% 552569fe-c6df-4edb-a553–9efdcf682fb3 Rack1-K8Demo\n UN 10.2.2.4 169.7 KiB 32 42.5% 92060271-d8cd-48be-a489-c830a8553462 Rack1-K8Demo\n UN 10.2.3.9 90.81 KiB 32 54.1% 47e8c9e2-a6d9–4276–88ae-6fe2256ca2af Rack1-K8Demo\n\nYou will need to wait for the status of the nodes to be Up and Normal (UN) to execute the commands in the next steps.\n\n#### Troubleshooting\n\nIf your Cassandra instance is not running properly, you may check the logs using the command below. Replace with your pod name. For example, `cassandra-0`:\n\n kubectl logs \n\nIf your Cassandra nodes are not joining, delete your controller/statefulset then delete your Cassandra service:\n\n kubectl delete statefulset cassandra\n\nIf you created the Cassandra StatefulSet:\n\n kubectl delete svc cassandra\n\nTo delete everything:\n\n kubectl delete statefulset,pvc,pv,svc -l app=cassandra\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-deploy-a-cassandra-cluster-ring-on-kubernetes-openebs/)*.\n","notHasFeatureImage":false,"slug":"how-to-deploy-a-cassandra-cluster/ring-on-kubernetes-+-openebs"},{"id":112,"title":"How to install OpenEBS on IBM Cloud Private","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"19-11-2017","tags":["Helm Charts","OpenEBS","Solutions","ICP"],"excerpt":"What is IBM Cloud Private? IBM Cloud Private (ICP) is a new application platform that is based on Kubernetes and provides services for developing and managing on-premises containerized applications. ","content":"\n## What is IBM Cloud Private?\n\n**IBM Cloud Private (ICP)** is a new application platform that is based on **Kubernetes** and provides services for developing and managing **on-premises** containerized applications. ICP Community Edition *(ICP-CE)* is distributed free of charge for non-production use and is available on Docker Hub. For commercial use, you would need the Enterprise package.\n\nIn my previous blog post, [Introduction to IBM Cloud Private](http://containerized.me/introduction-to-ibm-cloud-private/), I have covered step-by-step installation of ICP 2.1. This time I will focus on configuring **OpenEBS** as a **persistent storage** option and deploying a stateful workload (MongoDB) using OpenEBS storage classes.\n\n## Prerequisites\n\n### Hardware\n\n- Minimum three x64 servers\n\n### Software\n\n- [Ubuntu Server 16.04 LTS](https://www.ubuntu.com/download/server)\n- IBM Cloud Private 2.1\n- [OpenEBS](https://github.com/openebs/openebs)\n\n### Install IBM Cloud Private\n\nFollow instructions from [Introduction to IBM Cloud Private](http://containerized.me/introduction-to-ibm-cloud-private/) to deploy a multi-node ICP cluster.\n\n### Install OpenEBS on ICP\n\n1. Log in to the ICP console and go to the **Admin/Repositories** menu.\n\n![OpenEBS on ICP](https://cdn-images-1.medium.com/max/800/0*PPZPNSr9_mW_9AZq.png)\n\n1. Click **Add repository**.\n\n![Add repository](https://cdn-images-1.medium.com/max/800/0*ZNaLIkk1gxFLWUJK.png)\n\n1. Add a chart repository with the following parameters:\n — **Name:** openebs-charts\n — **URL:** [https://openebs.github.io/charts/](https://openebs.github.io/charts/)\n\n![Confirm charts](https://cdn-images-1.medium.com/max/800/0*2m2J6V9YhnYk5_Cx.png)\n\n1. After you click **Add**, confirm that **openebs-charts** is listed under Repositories.\n\n![Catalog menu](https://cdn-images-1.medium.com/max/800/0*wkPxIB_Q2DevkgWh.png)\n\n1. Go to the **Catalog** menu, select **openebs** from the list.\n\n![Configure](https://cdn-images-1.medium.com/max/800/0*7Lt6IE4f_da0jZEB.png)\n\n1. On OpenEBS chart instructions page, click **Configure**.\n2. Configure OpenEBS deployment with the following parameters:\n — **Release name:** openebs- (you need to pick a unique name)\n — **Target Namespace:** default (namespace should be the same as your workload)\n — **rbacEnable:** true\n — **image pullPolicy:** IfNotPresent\n — **apiserver image:** openebs/m-apiserver\n — **apiserver tag:** 0.4.0\n — **provisione image:** openebs/openebs-k8s-provisioner\n — **provisioner tag:** 0.4.0\n — **jiva image:** openebs/jiva:0.4.0\n — **replicas:** 2 (Number of Jiva Volume replicas)\n\n![Installation](https://cdn-images-1.medium.com/max/800/0*qfLs4pg_3TE1PbCB.png)\n\n1. Click **Install**. When finished click **View Helm Release.**\n\n![Storage classes](https://cdn-images-1.medium.com/max/800/0*raLyHiJeZ0hC_BAk.png)\n\n1. On the Helm Release page, you can see the status of OpenEBS, deployment, and available **Storage Classes**.\n\n![Deploy stateful application](https://cdn-images-1.medium.com/max/800/0*-gCAd374s2jXY3AP.jpg)\n\n1. Now, let’s try to deploy a stateful app on OpenEBS.\n\n### Install MongoDB on OpenEBS\n\n1. Under **Catalog**, select **ibm-mongodb-dev** and click **Configure**.\n2. Configure MongoDB deployment with the following parameters:\n — **Release name:** mongodb- (you need to pick a unique name here)\n — **Target Namespace:** default (same as OpenEBS)\n — **persistence enabled:** true\n — **persistence useDynamicProvisioning:** true\n — **dataVolume storageClassName:** openebs-mongodb\n — **dataVolume size:** 2G (default is 20Gi, remove “i” — in current version it is not supported)\n — **database password:** mongo\n Accept the license agreements, keep all the other values as default and click **Install**.\n\n![Workloads release](https://cdn-images-1.medium.com/max/800/0*UTiLWk3zOy5bw_Wh.png)\n\n1. Go to **Workloads/Helm Releases** and select your MongoDB release. Under the **PersistentVolumeClaim** table you are going to see the volume claim and OpenEBS storage class.\n\n![Workloads deployment](https://cdn-images-1.medium.com/max/800/0*PNNp0nDxsZXzYwIH.png)\n\n1. If you go to the **Workloads/Deployments** page, you can find the storage controller and two volume replicas (as configured) running.\n\n![Repository](https://cdn-images-1.medium.com/max/800/0*uaEIPO8n2vY0yUet.png)\n\n1. Confirm that replicas are running on separate nodes. Click on the PVC name ending with **rep** (Example:pvc-23025190-c516–11e7-b45e-e8fd90000064-rep). Scroll down, and you will see that pods are running on separate hosts.\n\n![Deployment successful](https://cdn-images-1.medium.com/max/800/0*pD7rHAX_D8_cxcfl.png)\n\nYou have successfully deployed a stateful application on a persistent block storage presented by OpenEBS.\n\n### How does storage HA work for stateful workloads?\n\nHigh Availability storage (HA storage) is a storage system that is continuously operational. Redundancy is the key feature of HA storage as it allows data to be kept in more than one place while ensuring data protection and consistency.\n\nAn **OpenEBS Jiva Volume** is a controller deployed during the OpenEBS installation. Volume replicas are defined by the parameter we set above. The controller is an **iSCSI target** while the replicas play the role of a disk. The controller exposes the iSCSI target while the actual data is written. The controller and each replica run inside a dedicated container.\n\nAn OpenEBS Jiva Volume controller exists as a single instance, but there can be multiple instances of OpenEBS Jiva volume replicas. Persistent data is synchronized between replicas.\n\nOpenEBS Jiva Volume HA is based on various scenarios as explained in the following sections.\n\nNOTE: Each replica is scheduled in a unique K8s node, and a K8s node never has two replicas of one OpenEBS volume.\n\n### What happens when an OpenEBS volume controller pod crashes?\n\nKubernetes automatically re-schedules the controller as a new Kubernetes pod.\n Policies are in place that ensures faster rescheduling.\n\n### What happens when a K8s node that hosts OpenEBS volume controller goes offline?\n\nThe controller is automatically re-scheduled as a new Kubernetes pod.\n Policies are in place that ensures faster rescheduling.\n If Kubernetes node is unavailable, the controller gets scheduled on one of the available nodes.\n\n### What happens when an OpenEBS volume replica pod crashes for reasons other than node not-ready and node unreachable?\n\nThe replica is autoamtically re-scheduled as a new Kubernetes pod.\n The replica may or may not be re-scheduled on the same K8s node.\n There is data loss with this newly scheduled replica if it gets re-scheduled on a different K8s node.\n\n### What happens when a K8s node that hosts OpenEBS volume replica goes offline?\n\nThere is no storage downtime as the other available replica displays inputs/outputs.\n Policies are in place that does not allow re-scheduling of crashed replica (as the replica is tied to a node’s resources) to any other node.\n\n---\n\n*Originally published at *[*Containerized Me*](http://containerized.me/how-to-install-openebs-on-ibm-cloud-private/)*.*\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-ibm-cloud-private"},{"id":113,"title":"How to deploy Jenkins on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"16-11-2017","tags":["Jenkins","Kubernetes","Open Source","Solutions","OpenEBS"],"excerpt":"Modern development requires Continuous Integration / Continuous Delivery (CI/CD) and it means building and validating your software on every commit to make sure your development & test environments are always up-to-date. ","content":"\nModern development requires [Continuous Integration](https://aws.amazon.com/devops/continuous-integration/) / [Continuous Delivery](https://aws.amazon.com/devops/continuous-delivery/) (**CI/CD**) and it means building and validating your software on every commit to make sure your development & test environments are always up-to-date. This level of automation is a combination of cultural philosophies (aka **DevOps**) and practices. CI/CD increases an organization’s ability to deliver applications and services at high velocity. Jenkins serves as the **workflow engine** to manage this **CI/CD pipeline** from source to delivery.\n\nDeploying Jenkins on Kubernetes provides the following benefits:\n\n- Isolates different jobs from one another\n- Quickly clean a job’s workspace\n- Dynamically deploy or schedule jobs with Kubernetes pods\n- Allows increased resource utilization and efficiency\n- Dynamically scale up Jenkins slaves on demand\n\nEspecially, running dynamic slaves in a Kubernetes/Docker environment and automating the scaling of Jenkins slaves running in Kubernetes on top of OpenEBS can **minimize the deployment time and cost**. With OpenEBS, you can build extremely scalable test cycles. You will be able to create instant snapshots (thanks to the [**CoW**](https://en.wikipedia.org/wiki/Copy-on-write)) from the master and **deploy new slaves faster and dynamically on-demand**. This process will eliminate the need to perform container-to-container copies.\n\nIn Jenkins, slaves are optional. OpenEBS can also help when you have a smaller environment and running a **monolithic master**. In that model, state on the master would be lost when you shut down the Jenkins master service. When using monolithic master on OpenEBS, your volume is persistent and replicated over to n nodes *(defined in your OpenEBS storage class)*. In that case, the master can exit, even if your node fails it can start on other nodes, migrate from private to public cloud, vice-versa when needed and your data will follow you.\n\nLet’s deploy Jenkins on our existing K8s cluster with OpenEBS. You will notice that it’s not much different than deploying on local storage, except your data will be protected with OpenEBS.\n\n## Prerequisites\n\n### Software\n\n- [Docker](https://docs.docker.com/engine/installation/) installed\n- Kubernetes 1.7.3+ RBAC enabled cluster\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n## Deploy Jenkins Pod with Persistent Storage\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following simple steps to launch Jenkins service with a monolithic master.\n\nBefore getting started, check the status of the cluster using the following command.\n\n kubectl get nodes\n\nIn my environment, I have one master and two worker nodes.\n\n ubuntu@ip-172–23–1–115:~$ kubectl get nodes\n NAME STATUS ROLES AGE VERSION\n ip-172–23–1–115.us-west-2.compute.internal Ready master 2h v1.8.3\n ip-172–23–1–144.us-west-2.compute.internal Ready 2h v1.8.3\n ip-172–23–1–244.us-west-2.compute.internal Ready 2h v1.8.3\n\nDownload the `Jenkins.yml` file to your host, which has access to kubectl.\n\n wget https://raw.githubusercontent.com/openebs/openebs/master/k8s/demo/jenkins/jenkins.yml\n\nThis file looks like below. You can edit and specify a different OpenEBS storage class before you apply.\n\n kind: PersistentVolumeClaim\n apiVersion: v1\n metadata:\n name: jenkins-claim\n annotations:\n volume.beta.kubernetes.io/storage-class: openebs-standard\n spec:\n accessModes:\n — ReadWriteOnce\n resources:\n requests:\n storage: 5G\n — -\n apiVersion: extensions/v1beta1\n kind: Deployment\n metadata:\n name: jenkins\n spec:\n replicas: 1\n template:\n metadata:\n labels:\n app: jenkins-app\n spec:\n securityContext:\n fsGroup: 1000\n containers:\n — name: jenkins\n imagePullPolicy: IfNotPresent\n image: jenkins/jenkins:lts\n ports:\n — containerPort: 8080\n volumeMounts:\n — mountPath: /var/jenkins_home\n name: jenkins-home\n volumes:\n — name: jenkins-home\n persistentVolumeClaim:\n claimName: jenkins-claim\n — -\n apiVersion: v1\n kind: Service\n metadata:\n name: jenkins-svc\n spec:\n ports:\n — port: 80\n targetPort: 8080\n selector:\n app: jenkins-app\n type: NodePort\n\nNow apply `jenkins.yml` file.\n\n kubectl apply -f jenkins.yml\n\n![results](/images/blog/results.png)\n\nGet the status of running pods using the following command.\n\n kubectl get pods\n\nResult should like similar to below, and the Jenkins pod running.\n\n ubuntu@ip-172–23–1–115:~$ kubectl get pods\n NAME READY STATUS RESTARTS AGE\n jenkins-797b888448-pfx8x 1/1 Running 0 11m\n maya-apiserver-5994b58bbb-ck2tv 1/1 Running 0 2h\n openebs-provisioner-6f45dcf459-qjdlx 1/1 Running 0 2h\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-ctrl-864fcb6f74–2phfw 1/1 Running 0 11m\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-rep-575d85d96c-dk4dq 1/1 Running 0 11m\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-rep-575d85d96c-pzrgn 1/1 Running 0 11m\n\nAs you noticed, your OpenEBS controller `pvc-…-ctrl-…` and two copies of persistent volumes `pvc-…-rep-…` are also deployed and running.\n\nGet the status of underlying persistent volumes used by Jenkins deployment using the following command.\n\n kubectl get pvc\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl get pvc\n NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE\n jenkins-claim Bound pvc-94586807-cb09–11e7-b125–064dff6dc2a2 5G RWO openebs-standard 22m\n\nGet the status of Jenkins service using the following command:\n\n kubectl get svc\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl get svc\n NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\n jenkins-svc NodePort 10.3.0.17 80:31705/TCP 25m\n kubernetes ClusterIP 10.3.0.1 443/TCP 3h\n maya-apiserver-service ClusterIP 10.3.0.34 5656/TCP 3h\n pvc-94586807-cb09–11e7-b125–064dff6dc2a2-ctrl-svc ClusterIP 10.3.0.100 3260/TCP,9501/TCP 25m\n\n### Launching Jenkins\n\nThe Jenkins deployment YAML file `jenkins.yaml` we have used above creates a NodePort service type to make Jenkins available outside the cluster.\n\nGet the node IP Address that is running the Jenkins pod using the following command.\n\nNote: Replace your pod name with your the pod name returned when you ran `kubectl get pods` command.\n\n kubectl describe pod jenkins-797b888448-pfx8x | grep Node:\n\nExample output below:\n\n kubectl describe pod jenkins-797b888448-pfx8x | grep Node:\n Node: ip-172–23–1–144.us-west-2.compute.internal/172.23.1.144\n\nGet the port number from the Jenkins service using the following command:\n\n kubectl describe svc jenkins-svc | grep NodePort:\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl describe svc jenkins-svc | grep NodePort:\n NodePort: 31705/TCP\n\nIP above is your private IP on AWS, which can be used if you are accessing through another instance on AWS. To access it remotely, you also need to open that port on E2C instance’s security group.\n\nGo to the Network & Security -> Security Group settings in the left hand navigation\nFind the **Security Group** that your instance is a part of. Click on **Inbound Rules**. Click on **Edit** and **Add Rule** button. Then add HTTP (port 31705). Click **Save**.\n\nNow, combine your public IP and port number and open that in your browser. In my case, it is [https://34.223.235.50:31705.](https://34.223.235.50:31705.)\n\nOnce you access the URL the Getting Started page is displayed. Follow the procedure below to setup Jenkins.\n\nProvide the [cci]initialAdminPassword[/cci] in the Unlock Jenkins screen and copy the password in the [cci]Administrator password[/cci] field. Click **Continue**.\n\n![unclock jenkins](/images/blog/unlock-jenkins.png)\n\nGet the password using the following command:\n\n kubectl exec -it jenkins-797b888448-pfx8x cat /var/jenkins_home/secrets/initialAdminPassword\n\nExample output below:\n\n ubuntu@ip-172–23–1–115:~$ kubectl exec -it jenkins-797b888448-pfx8x cat /var/jenkins_home/secrets/initialAdminPassword\n 5aa044d226d1466eb84621e75e369c64\n\nOn the **Customize Jenkins** screen click on **Install suggested plugins**.\n\n![customize jenkins](/images/blog/customize-jenkins.png)\n\nConfigure the Administrator user in the **Create First Admin User** screen. Fill in the following fields.\n\n**Username:** — Key in the administrator username.\n**Password:** — Key in the password for the administrator.\n**Confirm password:** — Key in the password again and confirm.\n**Full name:** — Key in the administrator’s full name.\n\nClick **Continue as admin** if you want to perform further administrator tasks or click **Save and Finish**.\n You can now start using Jenkins!\n\n![jenkins-is-ready](/images/blog/jenkins-is-ready.png)\n\n\n_Originally published at [*Containerized Me*](http://containerized.me/how-to-deploy-jenkins-on-kubernetes-openebs/)_.\n","notHasFeatureImage":false,"slug":"how-to-deploy-jenkins-on-kubernetes-+-openebs"},{"id":114,"title":"How to deploy a PostgreSQL Cluster on Kubernetes + OpenEBS","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"02-11-2017","tags":["Crunchy","Kubectl","Solutions","Kubernetes","OpenEBS"],"excerpt":"Why Postgres on Kubernetes? Well, the answer is in the question. If you are already running Kubernetes on some form of cloud, you understand the ease-of-use, scalability, and monitoring benefits of Kubernetes that you can apply to your database at scale.","content":"\n## Why Postgres on Kubernetes?\n\nWell, the answer is in the question. If you are already running Kubernetes on some form of cloud, you understand the **ease-of-use**, **scalability**, and **monitoring** benefits of Kubernetes that you can apply to your database at scale.\n\nPostgreSQL is the **preferred** relational database for most developers around, although setting up a highly available Postgres cluster from scratch is always a challenge, being **cloud-native** adds a bit to the difficulty.\n\nThere are many ways to run **high availability** with PostgreSQL; for a list, see the [PostgreSQL Documentation](https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling). To be honest, manually setting it up is quite painful, while there are better ways available. My favorite **cloud-native** Postgres cluster deployment projects are [Crunchy Data](https://www.crunchydata.com/)’s, [Sorint.lab](https://www.sorint.it/)’s [Stolon](https://github.com/sorintlab/stolon) and [Zalando](https://jobs.zalando.com/tech/)’s [Patroni](https://github.com/zalando/patroni)/[Spilo](https://github.com/zalando/spilo).\n\nSince availability requires multi-node Kubernetes deployment instead of local minikube setup, I’ll deploy crunchy-postgres on my existing K8s cluster on AWS with two worker nodes. If you don’t have a Kubernetes cluster yet, see the [instructions to deploy one using StackPointCloud](http://containerized.me/how-to-install-openebs-on-aws-using-stackpointcloud/). Instructions after that are the same in any cloud or on-premises deployment.\n\n## Prerequisites\n\n### Software\n\n- [crunchy-postgres](https://hub.docker.com/r/crunchydata/crunchy-postgres/) (for cluster deployment)\n- [Docker](https://docs.docker.com/engine/installation/)installed\n- Kubernetes 1.5+ cluster installed\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed\n- [OpenEBS](https://github.com/openebs/openebs) installed\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account\n\n### Deploy Crunchy PostgreSQL cluster using kubectl\n\nOnce you have OpenEBS storage classes created on your K8s cluster, you can use the following simple steps to launch a highly available PostgreSQL service with one master and one replica.\n\nDownload the files to your host, which has access to kubectl\n\n cd $HOME\n git clone https://github.com/openebs/openebs.git\n cd openebs/k8s/demo/crunchy-postgres \n\n### Create the Stateful Set\n\nThe deployment will use the default images and credentials defined in the set.json file. To set custom users and passwords:\n\n vi ~/openebs/k8s/demo/crunchy-postgres/set.json\n\nJSON file should look like below, feel free to edit the number of replicas, credentials and storage capacity. Default uses the **openebs-standard** storage class, and it is 400M.\n\n {\n \"apiVersion\": \"apps/v1beta1\",\n \"kind\": \"StatefulSet\",\n \"metadata\": {\n \"name\": \"pgset\"\n },\n \"spec\": {\n \"serviceName\": \"pgset\",\n \"replicas\": 2,\n \"template\": {\n \"metadata\": {\n \"labels\": {\n \"app\": \"pgset\"\n }\n },\n \"spec\": {\n \"containers\": [\n {\n \"name\": \"pgset\",\n \"image\": \"crunchydata/crunchy-postgres:centos7–9.6–1.4.0\",\n \"ports\": [\n {\n \"containerPort\": 5432,\n \"name\": \"postgres\"\n }\n ],\n \"env\": [\n {\n \"name\": \"PG_MASTER_USER\",\n \"value\": \"master\"\n },\n {\n \"name\": \"PGHOST\",\n \"value\": \"/tmp\"\n },\n {\n \"name\": \"PG_MODE\",\n \"value\": \"master\"\n },\n {\n \"name\": \"PG_MASTER_PASSWORD\",\n \"value\": \"password\"\n },\n {\n \"name\": \"PG_USER\",\n \"value\": \"testuser\"\n },\n {\n \"name\": \"PG_PASSWORD\",\n \"value\": \"password\"\n },\n {\n \"name\": \"PG_DATABASE\",\n \"value\": \"userdb\"\n },\n {\n \"name\": \"PG_ROOT_PASSWORD\",\n \"value\": \"password\"\n }\n ],\n \"volumeMounts\": [\n {\n \"name\": \"pgdata\",\n \"mountPath\": \"/pgdata\",\n \"readOnly\": false\n }\n ]\n }\n ]\n }\n },\n \"volumeClaimTemplates\": [\n {\n \"metadata\": {\n \"name\": \"pgdata\"\n },\n \"spec\": {\n \"accessModes\": [\n \"ReadWriteOnce\"\n ],\n \"storageClassName\": \"openebs-standard\",\n \"resources\": {\n \"requests\": {\n \"storage\": \"400M\"\n }\n }\n }\n }\n ]\n }\n }\n\nSave the file and run the statefulset:\n\n ./run.sh\n\nThe above step will automatically create the OpenEBS volumes required for master and replica postgresql containers and few other Kubernetes objects:\n\n- Persistent Volumes (pvc-{UID1}, pvc-{UID2})\n- Persistent Volume Claim (pgdata-pgset-0, pgdata-pgset-1)\n- Replica Sets (pvc-{UID1}-ctrl-{random1},pvc-{UID1}-rep-{random2},pvc-{UID2}-ctrl-{random3},pvc-{UID4}-ctrl-{random4})\n- Service Account (pgset-sa)\n- Services (pgset, pgset-master, pgset-replica)\n- StatefulSet (pgset)\n- Pods (pgset-0, pgset-1)\n\n![GIF displaying OpenEBS volumes required for master and replica postgresql containers and few other Kubernetes objects](https://cdn-images-1.medium.com/max/800/0*_WTDmIAcGNUGL0zn.gif)\n\nThe volume details can be inspected using the standard kubectl commands. To check **persistent volume claims**:\n\n kubectl get pvc\n\n![Screenshot showing persistent volume claims](https://cdn-images-1.medium.com/max/800/0*Jj59F2CWdQqKOkjW.png)\n\nCheck **persistent volumes**:\n\n kubectl get pv\n\n![Screenshot showing persistent volumes](https://cdn-images-1.medium.com/max/800/0*cm0u7Ea_12FvQRC4.png)\n\nList the **services**, and you will see pgset, master and replica created:\n\n kubectl get service\n\n![Listing services](https://cdn-images-1.medium.com/max/800/0*d5PjsFswTOOSBAcq.png)\n\nList the **statefulsets**, and you will see pgset listed with two desired and current sets:\n\n![Listing statefulsets](https://cdn-images-1.medium.com/max/800/0*F3eKWl181xp3yKLJ.png)\n\nIf you use the **Kubernetes Dashboard**, you can see the same under **Workloads > Stateful Sets** and quickly scale up as well.\n\n![Kubernetes Dashboard](https://cdn-images-1.medium.com/max/800/0*fQO6h-cj00rbePIi.png)\n\n### Test your Database\n\nIf it is not installed previously, install psql client:\n\n sudo apt-get install postgresql-client\n\nTest the master as follows (default password is “password”, unless you changed it):\n\n psql -h pgset-master -U testuser password -c ‘table pg_stat_replication’\n\nAbove command should return output indicating that a single replica is connecting to the master.\n\nNow, test the replica as follows:\n\n psql -h pgset-replica -U testuser password -c ‘create table foo (id int)’\n\nThis command should fail as the replica is read-only within a PostgreSQL cluster.\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/)*.\n","notHasFeatureImage":false,"slug":"how-to-deploy-a-postgresql-cluster-on-kubernetes-+-openebs"},{"id":115,"title":"How to Install OpenEBS on AWS using StackPointCloud?","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"23-10-2017","tags":["Kubernetes","Solutions","StackPointCloud","Ubuntu","OpenEBS"],"excerpt":"What is StackPointCloud? StackPointCloud is a managed Kubernetes control plane to build cloud-native stacks on AWS, Google Cloud (GKE & GCE), Azure & DigitalOcean. ","content":"\n## What is StackPointCloud?\n\nStackPointCloud is a managed Kubernetes control plane to build cloud-native stacks on AWS, Google Cloud (GKE & GCE), Azure & DigitalOcean. StackPointCloud simplifies installation and aggregation of multiple Kubernetes clusters pretty much on any platform. Even if you are an expert, provisioning your own Kubernetes stack their easy to use interface and capabilities to centralize all your deployments in one place is compelling. StackPointCloud is free for the first 30 days and $49.95 a month after for any number of Kubernetes clusters.\n\n## Prerequisites\n\nMinimum requirements for deploying your Kubernetes clusters on StackPointCloud:\n\n### Hardware\n\n- None\n\n### Software\n\n- [OpenEBS](https://github.com/openebs/openebs)\n\n### Cloud Provider\n\n- [Amazon Web Services (AWS)](https://aws.amazon.com/) account (Other major providers supported by StackPoint, but not covered in this article)\n\n## Start your StackPoint Trial\n\nFirst, go to [stackpoint.io](https://stackpoint.io/) and click on the **Launch a Cluster** button to start your free trial.\n\n![Launch a Cluster button in stackpoint.io](https://cdn-images-1.medium.com/max/800/0*3Iro4mlPVlQolQfh.png)\n\nThen choose your cloud provider. In this example, I will use **AWS**.\n\n![Configure Access to AWS](https://cdn-images-1.medium.com/max/800/0*s0vkUYR7sJXoR6IU.png)\n\n#### Configure Access to AWS\n\nOn the next screen, we need to configure our provider. You need to provide AWS Access Key ID and Secret Access Key and optionally your SSH Key.\n\n![Cloud providers](https://cdn-images-1.medium.com/max/800/0*_2SUsICymTDtGlwK.png)\n\nIf you don’t know where to find them, follow the instructions [here](https://stackpointcloud.com/community/tutorial/how-to-create-auth-credentials-on-amazon-web-services-aws) to create your user.\n\nClick on **Add Credentials** button.\n\n![Add Credentials](https://cdn-images-1.medium.com/max/800/0*5LX2XDbBqhnm1au8.png)\n\nAfter you add your credentials, click on **Submit**.\n\n## Configure K8s Cluster\n\nOn the “Configure your cluster” page click the edit button on **Distribution** and choose **Ubuntu 16.04 LTS**.\n\n![Configure K8s Cluster](https://cdn-images-1.medium.com/max/800/0*ty0IA_1uuDxaCQoX.png)\n\nChange the **Cluster Name** something meaningful like **OpenEBS Demo**.\n\n![Change the Cluster Name](https://cdn-images-1.medium.com/max/800/0*50cyzQI-2DZIX-AG.png)\n\nI could separate my etcd into 3 nodes dedicated cluster, but for a functional demo hosting it on the same cluster works perfectly fine. You can leave all other options as default. Now click on **Submit** to create your cluster. This should take around 5–8 minutes to bring up one Master and two Workers Kubernetes Cluster.\n\n## Import OpenEBS Helm Charts\n\nClick on the **Solutions** tab on the top of the screen and select **Import Charts** from the upper left.\n\n![Import OpenEBS Helm Charts](https://cdn-images-1.medium.com/max/800/0*vZr9hqN35SCCsx-a.png)\n\nAdd the chart repo with the following details: \n — **name :** openebs-charts \n — **type :** packaged-charts \n — **repo url : **[https://openebs.github.io/charts/](https://openebs.github.io/charts/)\n\nClick on **Review Repository**.\n\n![Update chart repo](https://cdn-images-1.medium.com/max/800/0*lkT38CLmsESK2i1T.png)\n\nMake sure **Access Verified** shows ok and click on the **Save Repository** button to finish adding chart repo.\n\n![Save Repository](https://cdn-images-1.medium.com/max/800/0*tS9uArAROjoOLc05.png)\n\n## Adding OpenEBS to Your Kubernetes Cluster\n\nFirst, make sure your cluster and all nodes are up.\n\nOn the **Control Plane** tab click on your cluster name **OpenEBS Demo**.\n\n![Control Plane tab](https://cdn-images-1.medium.com/max/800/0*0wxTlbbO_yPMJZ8F.png)\n\nOnce the Kubernetes cluster is up on AWS with functional Helm, click on the **Solutions** tab and **Add Solution** button.\n\n![Solutions tab](https://cdn-images-1.medium.com/max/800/0*QofakUAHAb_DRYWp.png)\n\nAdd the solution with the following details:\n\n– **namespace :** default \n– **values -> rbacEnabled :** false \n\n![Install OpenEBS into your cluster](https://cdn-images-1.medium.com/max/800/0*JiSAsRHf5SND0Cbp.png)\n\nClick on **Install** to finally add OpenEBS into your cluster.\n\nState field should be green after OpenEBS is successfully added.\n\n![OpenEBS post install screenshot](https://cdn-images-1.medium.com/max/800/0*1nY357dtw3PNOfAi.png)\n\nNow your cluster is ready; you can run your workloads on openebs-standard storage class.\n\nTo confirm, click on **K8s Dashboard**. This will bring up your Kubernetes Dashboard UI in a new window. You should be able to find the **openebs-standard** option under **Storage Classes**.\n\n![ K8s Dashboard](https://cdn-images-1.medium.com/max/800/0*E5eYS81HcguHaG1r.png)\n\nI’ll cover some workload examples such as MongoDB, Percona, Cassandra, and [Postgres](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/) running OpenEBS on my next blogs (stay tuned).\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-install-openebs-on-aws-using-stackpointcloud/)*.\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-on-aws-using-stackpointcloud?"},{"id":116,"title":"How to Install OpenEBS with Kubernetes using MiniKube","author":"Murat Karslioglu","author_info":"VP @OpenEBS & @MayaData_Inc. Lives to innovate! Opinions my own!","date":"22-10-2017","tags":["Container","Docker","minikube","Kubernetes","Solutions","OpenEBS"],"excerpt":"Whether you are a newbie to Kubernetes looking for a small setup to start or a developer who uses Kubernetes on a daily basis, minikube is the tool that helps you quickly set up and run a Kubernetes environment locally. ","content":"\n## What is MiniKube?\n\nWhether you are a newbie to Kubernetes looking for a small setup to start or a developer who uses Kubernetes on a daily basis, minikube is the tool that helps you quickly set up and run a Kubernetes environment locally. minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.\n\nThere are several options available for developers to install minikube based on an operating system. You can read the detailed instructions for the three most popular operating systems in [minikube Setup](https://github.com/kubernetes/minikube).\n\nHowever, if you are already an experienced minikube user, skip the minikube setup instructions and jump directly to the **Setup OpenEBS** section.\n\nIn this post, I will explain how to set up Kubernetes using minikube directly on Ubuntu 16.04 (without using any VM drivers) and how to configure OpenEBS in hyper-converged mode or, more accurately, create your Container-Converged Infrastructure using OpenEBS Container Attached Storage (CAS).\n\n## Prerequisites\n\nMinimum requirements for minikube:\n\n### Hardware\n\n- Machine Type — minimum 4 vCPUs.\n- RAM — minimum 4 GB.\n- VT-x/AMD-v virtualization must be enabled in your system BIOS\n\n### Software\n\n- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n- If using macOS:\n- xhyve driver, [VirtualBox](https://www.virtualbox.org/wiki/Downloads), or VMware Fusion.\n- If using Linux:\n- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or KVM.\n\n**NOTE:** minikube supports the `-vm-driver=none` option that runs Kubernetes components on the host and not in a VM. Docker is required to use this driver, but no the hypervisor.\n\n- If using Windows:\n- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) or Hyper-V. VMware Workstation is not supported.\n\nSince VirtualBox is available on all three platforms, I will describe this option.\n\n## Install VirtualBox\n\nI will not cover the details of VirtualBox installation since it is very common and instructions are widely available online.\n\n1. Go to the [Virtualbox website](https://www.virtualbox.org/wiki/Downloads).\n2. Download and install the binaries required for your operating system.\n\nMake sure that you install [VirtualBox 5.2.0 Oracle VM VirtualBox Extension Pack](http://download.virtualbox.org/virtualbox/5.2.0/Oracle_VM_VirtualBox_Extension_Pack-5.2.0-118431.vbox-extpack) as well.\n\nWhen I was writing this blog post, the most current version was VirtualBox-5.2.0–118431.\n\nOnce VirtualBox is installed, you will see a screen similar to the following:\n\n![Post VirtualBox install screenshot](https://cdn-images-1.medium.com/max/800/0*HztM26xqSWKiYaIx.png)\n\n**NOTE:** You can also use KVM, Hyper-V, and VMware Fusion.\n\n## Install Ubuntu\n\nCreate a new VM with 4 vCPUs, 4Gb memory, and 10GB disk space.\n\n![Creating VM](https://cdn-images-1.medium.com/max/800/0*8wqBzAyAPf_LsbFk.png)\n\nDownload your preferred version of [Ubuntu](https://www.ubuntu.com/download). I will be using Ubuntu 16.04.3 LTS.\n\nUnder **VM Settings/Storage**, mount your ISO image and power on the VM.\n\nInstall Ubuntu with default options. I used *openebs/password* as username/password for simplicity. If you use something else make sure to replace it with yours when you follow the instructions.\n\nFinally login to your Ubuntu VM.\n\nOn your Ubuntu host, install the SSH server:\n\n sudo apt-get install openssh-server\n\nNow you should be able to access your VM using SSH. Check the status by running:\n\n sudo service ssh status\n\n![Accessing your VM using SSH](https://cdn-images-1.medium.com/max/800/0*1rUwIrG2T0EzoBJj.png)\n\nDisable firewall on your Ubuntu VM by running:\n\n sudo ufw disable\n \n\nInstall curl if it’s not already installed:\n\n sudo apt install curl\n\nBy default, for each virtual machine, VirtualBox creates a private network (10.0.2.x) which is connected to your laptop’s network using NAT. However, you may not be able to your VMs from your localhost through SSH just yet. To access your VM, you need to configure port forwarding. In the network setting of the VM. Click on **Advanced/Port Forwarding** and create a rule with the **Host port 3022 **and **Guest Port 22**. Name it *SSH* and leave other fields blank.\n\n![Configuring port forwarding](https://cdn-images-1.medium.com/max/800/0*uDKLTcZapcEZfK3E.png)\n\nNow you can connect to your Ubuntu VM from your laptop using SSH with localhost as the address and port 3022 instead of 22. Connect to your Ubuntu VM using the following credentials: `openebs/password`\n\n## Install Docker\n\nTo get the latest version of Docker, install it from the official Docker repository.\n\nOn your Ubuntu VM, run the following commands:\n\n curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n sudo add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable”\n sudo apt-get update\n\n![Install the latest version of Docker](https://cdn-images-1.medium.com/max/800/0*-4QRgWvjit9qyKaq.png)\n\nConfirm that you want to install the binaries from the Docker repository instead of the default Ubuntu repository by running:\n\n sudo apt-get install -y docker-ce\n\n![Install the binaries from the Docker repository](https://cdn-images-1.medium.com/max/800/0*Hh8nFvl7xArgJnN-.png)\n\nInstall Docker and make sure it’s up and running after installation is complete:\n\n sudo apt-get install -y docker-ce\n sudo systemctl status docker\n\n![Install Docker](https://cdn-images-1.medium.com/max/800/0*NTvaIXL4LiPakwEy.png)\n\n## Add iSCSI Support\n\nOpenEBS uses iSCSI to connect to the block volumes. Therefore, you need to install the `open-iscsi` package on your Ubuntu machine.\n\nOn your Ubuntu host, run:\n\n sudo apt-get update\n sudo apt-get install open-iscsi\n sudo service open-iscsi restart\n\n![Install the open-iscsi package on your Ubuntu machine](https://cdn-images-1.medium.com/max/800/0*OmIy-bxY3PrD_HYT.png)\n\nCheck that the iSCSI initiator name is configured:\n\n sudo cat /etc/iscsi/initiatorname.iscsi\n\nVerify the iSCSI service is up and running:\n\n sudo service open-iscsi status\n\n![Check that the iSCSI initiator name is configured](https://cdn-images-1.medium.com/max/800/0*30EupY6kOMa30SMj.png)\n\n## Set up minikube and kubectl\n\nOn your Ubuntu host, install minikube by running:\n\n curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\n chmod +x minikube\n sudo mv minikube /usr/local/bin/\n\n![On Ubuntu host, install minikub](https://cdn-images-1.medium.com/max/800/0*62DCuwG4tX8iU_AX.png)\n\nInstall kubectl:\n\n curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl\n chmod +x kubectl\n sudo mv kubectl /usr/local/bin/\n\n![Install kubectl](https://cdn-images-1.medium.com/max/800/0*9jZx-rusvrdn9mEe.png)\n\nSet up directories for storing minkube and kubectl configurations:\n\n mkdir $HOME/.kube || true touch $HOME/.kube/config\n\nSet up an environment for minikube by adding the following lines to the end of the `~/.profile` file:\n\n export MINIKUBE_WANTUPDATENOTIFICATION=false\n export MINIKUBE_WANTREPORTERRORPROMPT=false\n export MINIKUBE_HOME=$HOME\n export CHANGE_MINIKUBE_NONE_USER=true\n export KUBECONFIG=$HOME/.kube/config\n\nConfirm that environment variables are saved in your profile file:\n\n cat ~/.profile\n\n![Confirm that environment variables are saved in your profile file](https://cdn-images-1.medium.com/max/800/0*rxjoxM6qkYkppd5h.png)\n\nStart minikube:\n\n sudo -E minikube start — vm-driver=none\n\n![Start minikube](https://cdn-images-1.medium.com/max/800/0*UaQ_6Y2m4hv6P4oc.png)\n\nIf you forgot to install Docker, you will get the following error:\n\n![Error screenshot when forgot to install Docker](https://cdn-images-1.medium.com/max/800/0*ysp8RnG5DWDu_Q0j.png)\n\nWhen using the none driver, the kubectl config and credentials generated will be root-owned and will appear in the root home directory. To fix this, set the correct permissions:\n\n sudo chown -R $USER $HOME/.kube\n sudo chgrp -R $USER $HOME/.kube\n sudo chown -R $USER $HOME/.minikube\n sudo chgrp -R $USER $HOME/.minikube\n\n## Verify minikube configuration\n\nVerify that minikube is configured correctly and it has started by running:\n\n minikube status\n\n**Example:**\n\n![Verify minikube configuration](https://cdn-images-1.medium.com/max/800/0*yK3Wlyy81I15tNZp.png)\n\n**Note**\n\n- If the minikube status displays **Stopped**, add the `sudo minikube start` command.\n- If you forgot to set the permissions, minikube will display errors indicating permissions denied to configuration files, fix the permissions by running the following commands:\n\n sudo chown -R $USER $HOME/.kube\n sudo chgrp -R $USER $HOME/.kube\n sudo chown -R $USER $HOME/.minikube\n sudo chgrp -R $USER $HOME/.minikube\n\n## Verify Kubernetes configuration\n\nCheck that kubectl is configured and services are up and running by getting the list of Kubernetes nodes and pods:\n\n kubectl get nodes\n kubectl get pods — all-namespaces\n\n![Verify Kubernetes configuration](https://cdn-images-1.medium.com/max/800/0*noWgoiv0GLk43BRB.png)\n\n## Set up OpenEBS\n\nDownload the latest OpenEBS Operator files using the following commands:\n\n git clone https://github.com/openebs/openebs.git\n cd openebs/k8s\n\n![Download the latest OpenEBS Operator files](https://cdn-images-1.medium.com/max/800/0*UNKK2cZhYPJVbTDx.png)\n\nBy default, OpenEBS launches OpenEBS Volumes with two replicas. To set one replica, as is the case with a single-node Kubernetes cluster, in the openebs-operator.yaml file, specify the environment variable `DEFAULT_REPLICA_COUNT=1`. This is supported in OpenEBS version 0.4 onward.\n\n![Replica count screenshot](https://cdn-images-1.medium.com/max/800/0*SxXEzbDmpVA5ZhwS.png)\n\nApply the configuration changes:\n\n kubectl apply -f openebs-operator.yaml\n\n![Applying configuration change](https://cdn-images-1.medium.com/max/800/0*WB16UScHye4LClft.png)\n\nAdd the OpenEBS storage classes that can then be used by developers and applications:\n\n kubectl apply -f openebs-storageclasses.yaml\n\n![Adding the OpenEBS storage classes](https://cdn-images-1.medium.com/max/800/0*mojNYfZbll-g6bdk.png)\n\n#### Running stateful applications with OpenEBS storage\n\nTo use OpenEBS as persistent storage for your stateful workloads, set the storage class in the Persistent Volume Claim (PVC) of your application to one of the OpenEBS storage class.\n\nGet the list of storage classes using the following command. Choose the storage class that best suits your application.\n\n kubectl get sc\n\n![Getting the list of storage classes](https://cdn-images-1.medium.com/max/800/0*artfNnT8fZSziaKH.png)\n\nYou can find samples of YAML files for stateful workloads using OpenEBS under the `openebs/k8s/demo` folder.\n\n![Finding samples of YAML files for stateful workloads](https://cdn-images-1.medium.com/max/800/0*KCn5Z4-av-7Hevj_.png)\n\nNow you have your Kubernetes cluster up and running. In my next blog posts, I will cover the installation of stateful workloads such as Cassandra and [PostgreSQL](http://containerized.me/how-to-deploy-a-postgresql-cluster-on-kubernetes-openebs/), as well as the benefits of running your stateful workloads on OpenEBS. Stay tuned!\n\n---\n\n*Originally published at [Containerized Me](http://containerized.me/how-to-install-openebs-with-kubernetes-using-minikube/)*.\n","notHasFeatureImage":false,"slug":"how-to-install-openebs-with-kubernetes-using-minikube"},{"id":117,"title":"Why are users using OpenEBS *on* EBS?","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"19-10-2017","tags":["Container Native Storage","Featured","MySQL","Kubernetes","OpenEBS"],"excerpt":"We were a little surprised to see OpenEBS started to be used on top of EBS itself. So we dug in and asked why?","content":"\nWe were a little surprised to see OpenEBS started to be used on top of EBS itself. So we dug in and asked why?\n\nThe following lays out what we learned, focusing mostly on the most common use case.\n\nAs you likely know, broadly speaking, there are two types of stateful applications. We call them Mercy Apps and NoMercy apps (Not generic names, but we started using these names in the OpenEBS community discussions 🙂)\n\n1. **Mercy Apps** — The apps deal with data resiliency at the application level; for example, they synchronously replicate and disperse the data and will have less dependency on the high availability of storage underneath. Some examples of mercy apps are Cassandra and even (usually) MongoDB.\n2. **NoMercy Apps** — The apps that have a single copy of the data and have no idea of synchronous replication of data. These applications assume that the underlying storage is always (at least highly) available. The most famous example for NoMercy apps is the famous “MySQL server”.\n\nToday, most of the legacy apps are NoMercy Apps. Also, MySQL or Postgres tends to be the first choice for most of the developers and remains the most commonly used database for quick application development and deployment. The applications that use MySQL underneath are being containerized and probably being moved to the cloud container services such as Amazon ECS. It is important to note that even NoSql databases like MongoDB that protect the data across nodes are often deployed as a single copy (as a NoMercy App) and that we are seeing cases where users of NoSql use the storage for node pre-population for example instead of relying on node rebalancing at the application / NoSql level.\n\n![Shifting from legacy to microservices](https://cdn-images-1.medium.com/max/800/1*Bayd4nQST787TIbYo_5aWg.png) \n(***Shifting from legacy to microservices***)\n\nOnce you lift and shift your legacy app onto containers and the cloud, you ideally want to allow for Kubernetes to orchestrate the dynamism of container movements among various hosts; while this dynamism is core to the value of containerization, it also creates a unique challenge for the availability of data/storage to the application.\n\n![EBS volume may not be immediately available](https://cdn-images-1.medium.com/max/800/1*ISz4kvGREGlXZkBwiwSRjQ.png) \n(***EBS volume may not be immediately available***) \n\nAs the above drawing suggests, we learned from some OpenEBS community users that when app containers move from one host to another, the time it takes to detach the EBS volume from one host and to attach the same EBS volume to the new host can *cause downtime for the stateful application*.\n\n### There are two solutions to this problem.\n\n***First solution:*** Re-architect your application and make it a mercy application. Use the new age databases like Cassandra or MongoDB and configure them to protect the data at the application layer, which of course needs special training and quite a bit of work. Even the flavor of SQL support may change, so you may find yourself rewriting your queries, always a source of fun and enjoyment 🙂\n\n![Sync replicate at the DB level](https://cdn-images-1.medium.com/max/800/1*rdabUhTkx6iF3Ncv3EKlrQ.png) \n(***Sync replicate at the DB level***)\n\nBtw, if the thought of moving to a new Database is daunting, you would be interested in using plug-in code to the existing no-mercy apps to perform the synchronous replication. In the case of MySQL, one option is to migrate to MariaDB and then use the Galera sync replication plugin. You would be lucky if you already have a plugin that is resilient and doesn’t add performance overheads.\n\n***A second solution*** — that is probably the top reason users are running OpenEBS and similar containerized storage controllers ON EBS — is a more elegant one : *protect the data at the storage layer*. By putting OpenEBS into your pods as the provider of storage and letting it handle data placement on the local nodes (in this case one or more, yes, EBS volumes) while also replicating the data per the policies you prefer, you avoid rewiring your application.\n\n![Using OpenEBS for high availability of Mysql DB data on AWS EBS](https://cdn-images-1.medium.com/max/800/1*3npgXXxGEOFD4uh_KRvPng.png) \n(***Using OpenEBS for high availability of Mysql DB data on AWS EBS***)\n\nOk, great, however — you are now doing storage differently. Isn’t that itself a challenge?\n\nWell, if you are already moving towards Kubernetes then you are already learning the skills needed to run OpenEBS. OpenEBS is integrated into the Kubernetes storage architecture to make the volume provisioning basically the same behind the scenes experience as attaching the EBS volume for a given pod. Instead of attaching the AWS EBS volume to the application pod using kubernetes.io/aws-ebs provisioner, the developer simply uses kubernetes.io/openebs provisioner. The underlying pieces such as integrating the AWS EBS volume into OpenEBS volume are handled by the OpenEBS provisioner.\n\nMileage may of course vary. We are building OpenEBS in the open in part to get feedback from users like those that prompted me to write this blog in the first place. Please share with the community experience of running OpenEBS on your Kubernetes based AWS ECS — whether your applications are Mercy or NoMercy. You can find the instructions and help [here](http://openebs.readthedocs.io/en/latest/install/cloud_solutions.html#amazon-cloud). And — yes — we welcome other bloggers 🙂 We’ve seen a huge ramp in issues and PRs and so forth in the last couple of months. The next step might be user blogging — please feel free and we’ll help if useful.\n","notHasFeatureImage":true,"slug":"why-are-users-using-openebs-*on*-ebs?"},{"id":118,"title":"How to build an e2e?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"17-10-2017","tags":["Cloud Storage","E2e Testing","Featured","Kubernetes","Storage"],"excerpt":"e2e which expands into end to end speaks for itself. It can be treated as component testing, integration testing, or something that tries to test stuff outside the scope of unit tests.","content":"\n### What is e2e?\n\nWell *e2e* which expands into end to end speaks for itself. It can be treated as component testing, integration testing, or something that tries to test stuff outside the scope of unit tests. I got attracted to this side of development when I was surprised looking at one of the smallest possible directory names ever in a code-based project. As I speak, we get to see a number of Go based projects having an *e2e* folder. I believe, it has been popularized by the likes of etcd, Kubernetes, and perhaps a few other open source communities.\n\n### How to build one?\n\nBefore even getting into the How’s part, we must rather be comfortable with following questions:\n\n— *Do we need to?* \n — *Does our project need one?*\n\nMore often than not, a code based project will love to have one. Perhaps not with this name but the need will definitely exist. Some of the guiding principles highlighted in this article will also help us in getting the answers to these questions. So keep reading.\n\n### Assuming our project needs one, how to build it then?\n\nAre there any guidelines/rules that we can follow? Here they are:\n\n— ***`Refer and re-use wherever possible than build from scratch`*** is the golden rule.\n\n— ***`Start Small`***. In this context, it refers to '*not to gulp the entire code base and be very selective even to the point of refusing 90% of the code that is available for free*'. This is the learning I have had all these years as a programmer. This is essential if one wants to reach the milestones in time which will otherwise become a mirage. I have burnt my fingers multiple times with regards to this principle. The latest one was when I tried to reuse the entire Kubernetes e2e for [Maya’s](https://github.com/openebs/maya/) e2e. Needless to say I failed miserably. Some of these reasons are explained in this article.\n\n### Is that all?\n\nDefinitely not. We will soon get into the How’s part which in turn is a repository of queries, doubts, and concerns.\n\nTo clarify it further, I had these queries when I started with the e2e journey for the [Maya](https://github.com/openebs/maya/) project.\n\n- Should e2e be shell wrappers over [CLI](https://en.wikipedia.org/wiki/Command-line_interface), for example, [kubectl](https://kubernetes.io/docs/user-guide/kubectl-overview/) & [mayactl](https://github.com/openebs/maya/tree/master/cmd/mayactl)?\n- Should it be tied to Go [*testing*](https://golang.org/pkg/testing/) library? Any benefits?\n- Should it use the [*Ginkgo*](https://onsi.github.io/ginkgo/) library? Anything to gain?\n- Should it be moulded with [*Ansible*](https://www.ansible.com/) and let the scene get enacted by its players (*read playbooks*)?\n\n### Simplicity — One Rule to Rule them All\n\nAll these queries should be answered with simplicity in mind. In other words, how to construct test code that is simple?\n\nWith simplicity in mind, let us pen down some dos and don’ts that will be an\nindicator of simplicity versus complexity.\n\n### Simplicity — Direct Thinking!!\n\nListed are some of the direct modes of thinking with regards to simplicity.\n\n- Code should just try to eliminate the repeating tasks of the developer, tester, or operator. It will be simpler if it builds logic on only those ingredients that play an inhibiting role in manual testing.\n- Find out one tool (*in other words a dependency*) that can make this test code easy to reason and comprehend. Build the code around this dependency. Though this dependency becomes a hard requirement, we are still good if it satisfies the simplicity rule.\n- It should be limited in its scope. For example, in the case of [Maya](https://github.com/openebs/maya/), its e2e is scoped to Kubernetes. Maya e2e avoids [*Ginkgo*](https://onsi.github.io/ginkgo/) etc libraries. It also avoids *Ansible* as the latter is not at all required.\n- It should be built using a high level language. Maya e2e uses Go as its only programming language. It does not use shell or any other scripts. This choice is also dependent on the scope the test code is targeting at. In the case of Maya, [Kubernetes](https://kubernetes.io/) (*and host of other container orchestrators*) use Go as their primary language. This helps Maya to abide by the golden rules mentioned earlier.\n\n### Simplicity — Inverted Thinking!!\n\nListed are some of the inverted modes of thinking with regards to simplicity.\n\n- It should not try to be another [DSL](https://en.wikipedia.org/wiki/Domain-specific_language) in the making. In addition, it should not deal with some smart syntax. It will instantly seem smart to the eyes which developed it, but will definitely repel others.\n- It should not get into the way of the developer or the tester in form of auto-generation of test code etc. This will lead to its brittleness.\n- It should not dedicate its logic to concurrency while running the test cases. It should not build its logic around CPU cores and parallelism for running the test cases either. It might seem concurrency is a required feature for a particular test case. However, do we take concurrency into factor while executing these cases manually? The test logic can have the concurrency built later in an iterative fashion. Remember “*shipping is better than perfect”* in these cases.\n- It should not play around with the abstraction of containers and their orchestrators. Remember this is not the reason for its existence in the first place. It goes against the prescription of simplicity. Hence, cut this noise if they are not a natural fit.\n- It should not mandate running the tests in containers. It is considered cool these days to run test cases in containers. However, it can backfire. For example, running your app and test cases as containers within a namespace or inside the same cluster. We do not want *100s* and *1000s* of containers getting spawned in the same setup that is meant to test your app which is again a set of containers.\n- It should not involve a learning curve. It should be an involving library that can take inputs from the team (*internal as well as external*) and evolve.\n- It `need not / must not` follow design patterns. The best it can do is to adhere to the core language’s best practices.\n\n### What else should the e2e try to achieve?\n\nAnswers to these questions will help you in getting the other aspects of your e2e.\n\n- Did we avoid fancy scripting ?\n- Is e2e (*especially the reusable code pieces*) better than the one it was created from?\n- Will *`anybody be able to contribute to anything`* in this e2e project?\n- Does it look similar to the practices followed by its programming language? Why? This leads to a 0 learning curve and hence is simpler to understand?\n- Do you still remember the `Golden Rules`? Add another one & that is **`End Small`**.\n\n*Thanks to Madhuri Hebbar and Uday Kiran.*\n","notHasFeatureImage":true,"slug":"how-to-build-an-e2e?"},{"id":119,"title":"Quick update on OpenEBS v0.4 — a developer friendly release!!","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"03-10-2017","tags":["Container Native Storage","Container Orchestration","minikube","Open Source","Updates","OpenEBS"],"excerpt":"OpenEBS v0.4.0 is out and I take pride in sharing that it is powered by the same set of tools, used by communities that have adopted DevOps DNA for delivering products.","content":"\n[OpenEBS v0.4.0](https://github.com/openebs/openebs/releases/tag/v0.4.0) is out and I take pride in sharing that it is powered by the same set of tools, used by communities that have adopted DevOps DNA for delivering products.\n\nPersonally, for me, the most exciting part of this release is that it involves contributions from community members across the globe. We have crossed more than 100 Pull Requests from 25+ contributors.\n\nOur decision to remain OpenSource is paying off!! As new contributors come on board, the following DevOps tools are helping us to sustain the quality:\n\n- [Github ](https://github.com/openebs/openebs/issues)for collaboration — managing code reviews, releases, and now project management (issues and milestones).\n- [Travis ](https://travis-ci.org/openebs/) and [DockerHub ](https://hub.docker.com/r/openebs/)— validate the code commits and release new docker images\n- [Jenkins and Ansible](https://github.com/openebs/openebs/tree/master/e2e) — run the e2e tests on the new docker images.\n- Static Analysis is provided through a wide range of tools like gofmt, flake8, codecov, goreport, and the list needs to expand.\n- [ReadTheDocs ](http://openebs.readthedocs.io/en/latest/index.html) — updating the live documentation site\n\nOpenEBS, like other storage options (EBS, Rook, and others), is already deeply integrated with Kubernetes and is now part of the [kubernetes-incubator project](https://github.com/kubernetes-incubator/external-storage/tree/master/openebs). OpenEBS also provides similar intuitive mechanisms to provide block storage to your stateful application on Kubernetes using concepts like:\n\n- StorageClasses\n- PersistentVolumeClaims\n- PersistentVolumes\n- DynamicProvisioner\n\nIn addition, unlike others, OpenEBS delivers container-native storage by *using Kubernetes (as opposed to running on Kubernetes)* itself as the underlying framework for scheduling and storing configuration data. There are also efforts underway for the upcoming release to make use of Kubernetes *LocalStorageManager, Kube-Dashboard, and CNCF projects like Prometheus, FluentD, Grafana, Jaegar*, etc., for managing and monitoring the storage functionality.\n\nYou can easily get OpenEBS running on your Kubernetes Cluster with the following two commands and then point your application’s PVC to one of the OpenEBS Storage Classes.\n\n`kubectl apply -f openebs-operator.yaml` \n`kubectl apply -f openebs-storageclasses.yaml`\n\nThe above YAML files can be downloaded from [here](https://github.com/openebs/openebs/tree/master/k8s). For detailed instructions refer to our [quick start guide](http://openebs.readthedocs.io/en/latest/getting_started/quick_install.html) or checkout our [sample stateful applications](http://openebs.readthedocs.io/en/latest/install/install_usecases.html) which include Percona, Jupyter, Postgresql, etc.,\n\nWhile you are at the documentation, you can also check out the additional deployment options that we have added with this release:\n\n- [Running Kubernetes and OpenEBS on AWS](http://openebs.readthedocs.io/en/latest/install/cloud_solutions.html)\n- [Running Kubernetes within minikube](http://openebs.readthedocs.io/en/latest/install/dev_solutions.html)\n\nThough we are still in the first leg of our journey, with OpenEBS v0.4, you get usable container-native storage, with enterprise storage capabilities like detailed *Volume IO statistics* and *Snapshots*.\n\n*Please refer to the [CHANGELOG](http://openebs.readthedocs.io/en/latest/release_notes/releasenotes.html), for a summary of updates in v0.4 and [ISSUELOG](https://github.com/issues?q=user%3Aopenebs+and+is%3Apr+and+merged%3A%3E2017-06-23+sort%3Acreated-asc)for the list of Pull Requests.*\n\nAnd we are always looking for help from OpenSource savvy community members. You can contribute in several ways — take your pick from our growing [task list](https://github.com/openebs/openebs/labels). Join us on [#Slack](http://slack.openebs.io/).\n","notHasFeatureImage":false,"slug":"quick-update-on-openebs-v0.4-—-a-developer-friendly-release!!"},{"id":120,"title":"Not Yet Another Distributed Storage System","author":"Jeffry Molanus","author_info":"Jeffry is the CTO at MayaData. At MayaData, his primary focus is to make sure the product is flexible and scalable. When he is not working with code, he practices martial arts.","date":"10-09-2017","tags":["Kubernetes","Docker","Container"],"excerpt":"These days, it seems that a lot of storage vendors are taking a scale out approach to delivering high-performance storage to meet the increasing demand for IOPS and bandwidth.","content":"\nThese days, it seems that a lot of storage vendors are taking a scale-out approach to deliver high-performance storage to meet the increasing demand for IOPS and bandwidth. Reduced latency is also high on the storage requirement list, however, scale-out distributed systems typically result in the inverse, i.e., it increases latency.\n\nAlthough complex distributed systems are easier to build these days due to a variety of factors including maturing software implementations of the likes of Paxos, distributed hash tables, and RAFT it seems that creating a storage system that utilizes these concepts and is easy to manage and maintain in production — is not.\n\nOn the other hand, if you want to leverage the distributed nature of storage at the client level, more often than not, the end-users are required to run specialized clients/drivers to unleash the enormous bandwidth these systems can deliver — and so now your storage has infected your client which makes it even less attractive.\n\nArguably, if you need hundreds of gigabytes of throughput for a particular workload, the Linux kernel comes preloaded with [one](http://www.orangefs.org/). So you have to look no further from a tech support side of things — as what better experts to find that concern themselves with the Linux kernel and thus OrangeFS, right? No need to search any further, right?\n\nLet’s first try to understand why a lot of people I’ve spoken to in the past typically chose **“scale-out.”** It seems they all like the idea of “**add another box”** to add performance and capacity. The fact that you can’t scale performance decoupled from capacity is what they take for granted. However, due to economics forcing IT segments to do things cheaper and more predictably, this is not the case — anymore.\n\nSome storage vendors have found a solution for this, by not selling you expensive boxes but rather just the software. You simply scale in any cloud on any hardware or so they say. However, this **“any any”** approach does not fit in with the requirement to make things more predictable in fact quite the opposite.\n\nSo if we summarize the downsides:\n\n- Distributed storage difficult to develop and is hard and nasty to manage in production\n- Specialized drivers needed to unleash the real potential\n- Scaling the number of nodes does not decrease latency, in fact, usually the opposite (depending on implementation)\n- Best scales bandwidth and IOPS, however, this is not what a typical workload requires\n- Complex consistency models create surprises regarding what is on disk or not\n- Not especially good at leveraging flash or NVMe to deliver great performance and a small footprint\n- Big blast radius — the more data you can put in one system, the more you might lose or at least lose access to when you most need it\n- The complexity and need for quick metadata updates across nodes both argue against multi-cloud deployments (though there are some scale-out file systems being built that claim to address these issues)\n\nNow there will always be vendors out there that claim to have solved it all, unlimited scale-out, never ending IOPS, more bandwidth than the whole of the internet combined, and latency — sure we do that too…\n\nAt [OpenEBS](https://www.openebs.io/), however, we took a different approach by not trying to solve the distributed problems but to take a step back and try to determine what are the real problems people need to solve?\n\nSpeaking to our early tech-preview customers, we were shocked with awe to see that due to the complexity of storage these days, they simply revert to [Direct Attached Storage](https://en.wikipedia.org/wiki/Direct-attached_storage) (DAS). “It won’t go any quicker than that”, they say. And you have to make a very, very, strong case to argue against that. In fact, come to think of it, it is nearly impossible if you consider the speed of NMVe devices.\n\nHas the storage market become so consumed with itself that it keeps making storage products so complex, with each vendor, having its own **“if — and — or buts”** that they are fed up with it and revert back to DAS? Did it become so unpredictable? The time of consolidated storage churn?\n\nYes, the storage market is consumed with itself. But there is more, the storage **needs** have also changed. In the early SAN days, it was about consolidating islands of storage into a bigger one as it would be easier to manage and the performance of the SAN would be higher than some individual devices. Virtual machines made it possible to consolidate compute which also made a lot of sense as boxes were mostly idle.\n\nHowever, all of this work was done for one reason and one only — to accommodate the piece of software that matters the most: the app.\n\nThe app is central and is the only thing that matters; everything around it is inflicted upon us.\n\nNobody wakes up one day and says, “I need to get myself a SAN” — it was likely the best of the worst options at that time.\n\nAs DevOps happened alongside containerization, the application itself has become a distributed system. A distributed system in the sense that subcomponents of the app as a whole may run on the same box, different box, and are loosely coupled by APIs one way or the other and working together to solve a complex (business) problem.\n\nAs applications have become distributed systems themselves, they have become easier to scale and thus don’t always require high IOPS low latency storage devices to scale performance-wise. So storage and capacity are now loosely coupled and suffer from data gravity in a different way than they once did. Additionally, data availability does not solely depend anymore on expensive storage arrays as applications are designed to replicate their data straight out of the GIT. As these apps are distributed by nature, we think that a distributed storage system is not only complex and nasty but completely unneeded.\n\nSo — what’s a storage vendor to do? Well, we have it easier, as we don’t have much legacy — though we do have lots of experience building storage on containers as a part of our [ElastiStor](http://www.cloudbyte.com/products/elastistor-os/) product. And so we were able to start with the customer questions discussed above — in the age of microservices and cloud and containers, what job are they looking to do regarding serving, moving, and protecting their data? And we can answer those questions in a way that is entirely free from whether it is storage strategy A, B, or C. In future blogs we’ll talk about OpenEBS more (of course) — and maybe more importantly we would like to discuss with you how we think the job storage is being asked to do has changed and hence, how and why the old storage industry may be coming to an end.\n\nPlease, feel free to join us on [slack](http://slack.openebs.io/) to discuss in a more real-time fashion.\n","notHasFeatureImage":true,"slug":"not-yet-another-distributed-storage-system"},{"id":121,"title":"ARA - Recording Ansible Playbook Runs","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"31-08-2017","tags":["Ansible","Automation","DevOps","OpenEBS"],"excerpt":"Ansible playbooks can generate quite a lot of console data. Add the -v (I do it by default!) and you have quite a lot to scroll through.","content":"\nAnsible playbooks can generate quite a lot of console data. Add the -v (I do it by default!) and you have quite a lot to scroll through. Oftentimes, one tends to feel the need for a better reporting mechanism — one which is easy to scour for specific task status, whilst having the luxury to extract additional debug info for the same if needed. Also, it would be great if this aid is available for playbook runs across time, i.e., for older playbook runs as well. This would be especially beneficial when running ansible-based CI suites, like @OpenEBS, where the application deployments, test setup & execution are driven by Ansible.\n\nSounds like a perfect requirement for a UI-based solution, doesn’t it?\n\nA lot of people using Ansible for medium-large scale deployments are known to use **Tower**, **Rundeck**, or the opensource alternative **semaphore** to achieve this (Tower has a lifelong self-support trial license without a few features for under 10 nodes). There is also **Foreman**, which is great in environments where a hybrid tool set — a puppet with ansible, or chef with ansible, etc., is used. Most of these are workflow management tools that do more than *just* what we desired above, i.e, playbook recording. These tools provide a centralized management capability wherein inventory, users, task/play scheduling, notifications can all be controlled from a dashboard. Now, it is also possible to integrate these with other popular CI tools like Jenkins (watch [this](https://www.youtube.com/watch?v=CqjeIiHvy30&feature=youtu.be)).\n\nHowever, if there is already a system in place to perform some of the things these tools do (dynamic inventories, playbook triggers/scheduling) while not really needing others (access control, graphs), and you are only looking for the ability to store and analyze playbook runs via UI, then **ARA** (Ansible Run Analysis) is your tool.\n\n![ARA (Ansible Run Analysis)](/images/blog/ansible-run-analysis.png)\n\nHeavily used by the OpenStack community in their CI projects, ARA is built to just *\"record\"* playbook runs (Read ARA’s [manifesto](http://ara.readthedocs.io/en/latest/manifesto.html#manifesto), to understand more about its narrow focus). It does this via an ansible callback plugin to store run details into a database and a web interface to visualize the database.\n\n![Viewing playbook tasks summary in ARA](/images/blog/playbook-tasks-summary-in-ara.png) \n(***Viewing playbook tasks summary in ARA!***)\n\n![Viewing task details in ARA](/images/blog/viewing-tasks-details-in-ara.png) \n(***Viewing task details in ARA***)\n\nWhile it uses SQLite and an embedded webserver, respectively, for these purposes, you could even customize it to use MySQL & Apache. The UI includes nifty features like host fact lookup, playbook params page, search filter, property based sort, and link to code snippets!\n\n![ARA code snippet](/images/blog/see-specific-task-ran.png) \n(***Click on action to see where the specific task ran***)\n\nA nice video explaining the web interface, with playbooks from the OpenStack-Ansible project is [here](https://www.youtube.com/watch?v=k3i8VPCanGo)\n\nIn addition to these, ARA also provides\n\n- Couple of Ansible modules for persisting & viewing some user data that one may want to view on the browser-based UI (like an ansible “fact”, but for visualization purposes :P).\n- A CLI to query the database (While I haven’t found much use for it till now, it is useful to custom create some reports)\n\nARA follows the same support cycle as the upstream Ansible community and is under active development (See [github](https://github.com/openstack/ara))\n\nHere is an [ansible role](https://github.com/openebs/openebs/tree/master/e2e/ansible/roles/ara) we have written that you could use to quickly setup ARA on your ubuntu box 🙂\n\nWe @OpenEBS, have found great benefit in using this tool and would happily recommend it for the use cases discussed. !!\n","notHasFeatureImage":true,"slug":"ara---recording-ansible-playbook-runs"},{"id":122,"title":"Ansible @ OpenEBS — The whys and hows","author":"Karthik Satchitanand","author_info":"Karthik has been into the Design and Development of tools for infrastructure as code, software testing performance & benchmarking & chaos engineering.","date":"31-07-2017","tags":["Ansible","Ci","DevOps","Jenkins","Vagrant"],"excerpt":"We are using Ansible as one of the critical moving parts of our automated test suite in the CI pipeline @OpenEBS. The question was expected in some ways,","content":"\nDuring a telephone conversation with a former colleague and good friend I was confronted with an interesting question, “I know Ansible is a great configuration management tool, why bend it as a test automation framework?”\n\nWe are using Ansible as one of the critical moving parts of our automated test suite in the CI pipeline @OpenEBS. The question was expected in some ways, what with the friend having spent a few good years working with more “traditional”, proprietary, and hand-built-from-scratch test-automation frameworks based on Perl. The subsequent discussion (mostly answers and follow-up questions) helped me internalize why we chose Ansible at OpenEBS and how better to use it.\n\nFelt this warranted a blog post to make my thoughts public and hey, of course, gain more feedback!\n\nOK, that let the cat out of the bag\n\n## Infrastructure as Code (IaC)\n\nOne of the biggest IT trends over the last few years has been managing infrastructure through automation. One might argue that puppet started way back in 2005, thereby making this practice far older than most believe, but the way it has taken ops departments of most organizations by storm in the past 5–6 years is nothing less than a revolution. In fact, the paradigm of DevOps is built on managing infrastructure as code. And when we say code, the expectation of most ops personnel around the “language” would be that it doesn’t require deep programming knowledge and have a steep learning curve — which is what DSL (Domain Specific Language, sometimes also referred to as Domain Scripting Language) based frameworks like ansible achieve. It also helps that ansible adopts an imperative programming model (using YAML) that works well because of its alignment with the traditional command-based approach of ops teams.\n\n![Ansible GitHub trends ](https://cdn-images-1.medium.com/max/800/1*7Di79EF1SxNqF0F0KD1E-A.jpeg)\n(***Ansible GitHub trends (Courtesy: [https://www.ansible.com/blog/another-good-year-for-ansible-users](https://www.ansible.com/blog/another-good-year-for-ansible-users))*** )\n\nBut, how does the above address our question?\n\n**Answer**: A major portion of the test duration of infrastructure-based software, such as storage software involves “manipulation” of infrastructure. Setting up bare-metal boxes, virtual machines, or containers, installing packages, executing various commands that control & alter system state, monitoring for specific behavior are key aspects of this process. Consider the need to run the above as batch processes and perform parallel execution on multiple nodes — and the inevitability of a workflow orchestrator dawns upon you. Especially so when you are testing a solution like OpenEBS that is designed to provide storage for DevOps use cases (read more about this [here](/blog/storage-infrastructure-as-code-using-openebs))\n\nIs not an approach (and the tool) soaked in “**devops-ness**” a pre-requisite to test the storage solution specifically designed for DevOps use cases 🙂 ?\n\n## Why Ansible, why not chef, puppet, salt, etc.?\n\n![Ansible](https://cdn-images-1.medium.com/max/800/0*NQOK_gId-YBZMe02.png) \n\nErr.., this seems to have been done to death on the internet. Yet, new posts on this topic seem to spring up everytime I look. So, without discussing the why not, let me touch upon the aspects about ansible that appealed most to us.\n\n**Powerful, yet very simple**: Ansible’s power comes from its simplicity. Under the hood, it is just a DSL for a task runner over a secure shell (ssh) with intuitive modules for achieving most (if not all) system functions. The soft learning curve in ansible is one of its major advantages over its rivals (*Ok, I said I won’t do the why-not, but there is a feeling that puppet, chef, etc., are over-designed for the jobs they do.*) In an open-source project like OpenEBS, a need to build a template for the contributors to write their own test workflows without spending too much time was an important consideration.\n\n**Idempotency**: The ansible playbook (a set of tasks written using the modules mentioned previously) when run twice gives the same end result. This is a great help when it comes to reusing testbeds.\n\n**Speed of execution**: One of the benefits of having an agent-less architecture (apart from a complexity-free install and usage experience), ansible playbooks zip through configuration, and other “system” tasks (mostly, test logic) fairly fast. While there are supposedly issues at scale (1000s of nodes), it works just great for our needs. Want to setup a working Kubernetes cluster with OpenEBS storage on-premise in less than 20 min? — check out the playbooks on our [github repo](https://github.com/openebs/openebs/blob/master/e2e/ansible/openebs-on-premise-deployment-guide.md)\n\n**Rich module library**: Ansible has modules for most things under the sun 😐 Nuff said, go look : [Ansible modules](http://docs.ansible.com/ansible/latest/modules_by_category.html)\n\n(*As an aside, this was one of the other reasons why we started using ansible as a test engine, apart from the infrastructure angle*)\n\n**Plugins**: While ansible is great for configuration management, workflow orchestration, etc., it needs to be able to work well with other tools/frameworks that make up the CI-CD pipeline( Jenkins, Vagrant, etc.,) And all these have ansible plugins (How much we use them is a topic for another day, but the point is ansible does have integration if you choose to utilize it). More important than existing integrations is the ease with which you can extend ansible’s capabilities with custom plugins. Python was a dev + ops favourite, even before DevOps became a fad and ansible is built using it -so, there you go!\n\n**Community**: One of the ansible’s biggest strengths is its community. We discussed extending ansible’s capabilities via custom plugins. Chances are you never have to write one, because the community already has two versions of it (*okay, I may be exaggerating*), but the active community makes it a lot easier to adopt. There is a lot of documentation available as well. All of this means, for most purposes, you don’t have to opt for paid support for issues OR add-ons until you really scale or get complex.\n\n## How are we using Ansible?\n\nAt OpenEBS, we are using Ansible as :\n\na) A means to enable rapid deployments of applications in user environments. Today, you can use our Ansible playbooks to get a Percona MySQL server instance or a PostgreSQL statefulset up and running with OpenEBS storage on-premise from plain vanilla VMs in double quick time, with single command execution. Even as I write this, efforts are underway to create playbooks to perform such deployments on the cloud, right from provisioning VM instances to running test loads to verify setup stability.\n\nb) As a “***test orchestrator***”, i.e., for provisioning testbeds, executing test logic, and notifying users. That doesn’t mean we have stopped writing shell scripts or python scripts in QA (there are still functions which one might have to execute via shell or the ansible “shell” module or python, due to lack of actual ansible modules. Not to mention existing scripts which one wouldn’t bother converting into playbooks).\n\nOur current CI workflow involves a Jenkins master polling for updates to git repos, followed by bringing up VMs on-premise using vagrant, configuring those using ansible, followed by execution of test playbooks and user notification on slack. The CI is still evolving and efforts are on to make it more robust — you could join the OpenEBS-CI [slack channel ](http://slack.openebs.io/) & browse the [github pages](https://github.com/openebs/openebs/tree/master/e2e) if you are an enthusiast/would like to contribute!\n\nIn forthcoming blogs, I would like to discuss more on the “***How***” and share thoughts, challenges, solutions around using ansible both as a preferred application deployment mechanism as well as a “test orchestrator”.\n\nThanks for reading !!\n","notHasFeatureImage":true,"slug":"ansible-@-openebs-—-the-whys-and-hows"},{"id":123,"title":"Are you afraid of Go?","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"27-07-2017","tags":["Docker","Go","OpenEBS","Programming Languages","Programming Tips","Tutorials"],"excerpt":"Are you an experienced C programmer or perhaps an adept Java hacker finding Go somewhat awkward to deal with.","content":"\nAre you an experienced **C** programmer or perhaps an adept **Java** hacker finding [**Go**](https://github.com/golang/go/wiki/whygo) somewhat awkward to deal with? There may be different elements of surprise or familiarity based on the programmer’s background e.g a Java programmer may relate the GOPATH settings with the settings associated with JAVA_HOME. On a similar note, the very same programmer might be surprised with Makefiles that are used to script the compilation, build & other stuff. I am sure though that this experience will be different for a seasoned C programmer.\n\nIf the above and perhaps multiple other reasons inhibit you from familiarising yourself with Go then you are not the only one. The best thing is there is a similar number of solutions that will help you play along with the Go pitch.\n\nGoing back in time, this thorny feeling was exactly what we had faced when we started [***OpenEBS***](http://openebs.io) development a year back. However one of the core goals of *OpenEBS* is ease of use; whether it is for the admin or the operator or the developer, tester, and so on. In other words, different personas involved during the Software Life cycle, Deployment & Maintenance should not go through a learning curve and rather experience *OpenEBS* as the simplest form of storage software that cuts through all the storage noise. This meant we did not want Go or any other programming language, as a matter of fact, to limit our way towards this enlightened journey.\n\nDuring those days we implemented some of the strategies that helped nullify the above pain points and provide a smooth cruise to all current & future contributors of *OpenEBS*. This article will focus on our strategies meant for the newbies trying to get their hands dirty with Go.\n\n***I have itemized our strategies as these steps:***\n\n1. Start your baby steps at GO [**Playground**](https://play.golang.org/). This will remove the initial hassles of downloading Go and setting up GOPATH & stimulate you towards coding (read *familiarising with the newer syntax*) & start believing the language by viewing & analyzing the output expected from your logic i.e. *WYSIWYG*.\n2. Get familiar with [**Vagrant**](https://www.vagrantup.com/) (*I understand this is a different tool & is in no way related to Go programming*). Vagrant with its pitch for ‘*Development Environments Made Easy*’ will help you to set up VMs which has all the necessary Go based downloads and GOPATH settings. One can search for simple open-source projects that does below:\n\n**a**.Vagrantfile that makes use of some sample Go programs (probably scripting a *git clone to *), &\n\n**b**. Makefiles that have the logic to compile & build.\n\nThere is one catch though! One has to install Vagrant & a preferred Hypervisor to enjoy the benefits of Step 2. So there is a bit of a learning curve involved.\n\n***However, this should not deter one to avoid Step 2, as it has huge benefits going forward. You will appreciate this step once you understand the difficulties in managing projects with their right versions and dependencies. A trivial mistake here will lead to bug injections. This step will rather help you enjoy the taste of CI right from your laptop.***\n\nAfter gathering enough confidence via these steps one can try out fancier stuff. Does ***running the builds in Travis*** or ***compiling the source code in a Docker container*** challenge you. You might as well explore some of this stuff & others in our [**tool-room**](https://github.com/openebs).\n\nThere may be various other options (*probably even simpler ones*) available in this '**DevOps**' world that are not mentioned here. E.g. [**GVM**](https://github.com/moovweb/gvm). However, I have tried to list down the game plans that have worked well for us. Do get back with your valuable insights on how you have succeeded to tame the initial apprehensions while trying out a new language.\n","notHasFeatureImage":true,"slug":"are-you-afraid-of-go?"},{"id":124,"title":"Tech Enthusiasts weighing in on Container Native Storage","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"27-07-2017","tags":["Docker","Game of Thrones","OpenEBS","Reddit","Statefulcontainers"],"excerpt":"These redditers are like the nights-watch-men (Operations Team) who are guarding the wall (production services) from army of dead (issues).","content":"\n*Full disclosure: I contribute to OpenEBS and I relish GoT (not IoT).*\n\n*This post is inspired by the *[*Reddit thread (r/Docker)*](https://www.reddit.com/r/docker/comments/6l0y3v/persistent_storage_with_docker_in_production/)* discussing implementation approaches for Stateful Containers.*\n\nThese redditers are like the nights-watch-men (Operations Team) who are guarding the wall (production services) from army of dead (issues). These redditers are akin to “the tech Enthusiasts” as depicted in the following diagram, who can foresee the challenges and are looking for innovative solutions. \n\n![Container native storage adaption curve](https://cdn-images-1.medium.com/max/800/1*11EOWUuoRjWn8pZ1uMXidg.png)\n\nWith S-less (server and state) architectures on the rise and faster networks at every possible endpoint, we can envision building and deploying services that can scale to yet unseen/unknown/unimaginable magnitude, **but!** (*Ned Stark once told me, that anything said before but*, ..*a GoT in-joke*) **State will be the bottleneck.**\n\nWhatever you do with Compute (S-less) containers, you always need to start and end with Data. Data is State. *State — is the beginning and end! the Alpha and Omega!*\n\nS-less will have no (busine$$) value without State! (*Well, agreed that you can offload the maintenance of State to someone else. But at what cost! Whoever owns the data, owns the $$*)\n\nThe real question is where to put the State!\n\n*Probably over-simplifying, but we can save the State using one of the following:*\n\n- *Connected Storage — Further divided into File/Block*\n- *Container Native Storage*\n\n**Connected Storage**: save the State external to container hosts and use the Orchestrator Volume Plugins to attach/detach storage from SAN/NAS or Cloud Disks.\n\nThere is a huge community (of cash rich vendors, a.k.a Roses, and Lions!) and a majority of enterprises (consumers of storage) locked in with these vendors are working on making this option succeed.\n\nThese enterprises are stuck with Roses or Lions not because they love them, but these are the only options that are available to them. This reverberates in the comments of the Reddit thread, here is a gist of the views expressed:\n\n(a) Connecting to NFS or EFS (from AWS)\n\nApplied Use Cases:\n\n- Applications that store data in the file — images or text or backup\n- Applications that need shared access to the data from multiple containers, running on different hosts.\n\nConcerns:\n\n- Hard to keep up with the performance and capacity needs of the massively scalable applications\n- Seeing Performance Issues — when saving a huge number of small-sized files\n- Need to be careful about Data Integrity from shared access. Needs a lot of hand-holding for secured access\n- Not a good option for databases or high random write workloads\n\n(b) Connecting to SAN or Cloud Disks (attached locally)\n\nApplied Use Cases:\n\n- Applications that are resilient to underlying storage failures like ElasticSearch/Cassandra\n- Databases like Percona/PostgreSQL\n\nConcerns:\n\n- Longer re-build times and degraded windows, as the size of the data increases.\n- Connecting one LUN per Container increases the boot-up times up to 10 minutes in worst cases\n\n*Roses and Lions have been there for a very long time, fighting for the dominion of the realms, by any means necessary. But these are summer lands. They are not prepared for the Great Winter that has begun in the North.*\n\nThe “tech Enthusiasts” (or the nights watch) have seen the challenges that the massively scaled applications (Winter) can bring and are seeking alternate options.\n\nA handful of vendors like [portworx](https://t.co/Aawo9fr4Dz), [storage_os](https://storageos.com/), [rook](https://rook.io/), [openebs](https://www.openebs.io/) are working on alternate options, what is now being termed as [Cloud Native Storage](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing) or [Container Native Storage](https://storageos.com/storageos-vision-cloud-native-storage-todays-modern/).\n\n*While the blogs above (and many others) dwell into what makes a storage container-native, the one that stands out is — ***the flexibility***.*\n\n*The tech enthusiasts/operations personnel will have the choice of technology used to deliver their services like Kubernetes, DockerSwarm, Mesos — on Google, Amazon, Azure, or Private Cloud with storage that integrates seamlessly with these cloud environments.*\n\n**Container Native Storage:** storage controller functionality is containerized and can co-exist with the containers (even fly with them) across the clouds.\n\nPortworx is leading the pack, has been successful in getting some reference customers (*we are yet to hear from them in the open forums — hard to convince the council at kings landing with just a reference!*)\n\nSome of the apprehensions surrounding this option are:\n\n- writing a new storage layer is hard.\n- missing some standard benchmarking tools that can clearly demonstrate the performance boost obtained by this relatively new way of provisioning storage.\n- there are questions raised about Rook / CEPH performance for DBs\n- dependency on the kernel drivers (this is probably hinting at PortWorx, if I read, between the lines/comments).\n\n*Winter is Here My Lord — We need new alliances!* We need to hear from more Operations and DevOps Personnel grappling with the storage issues.\n\nWe need them to spend more time towards sharpening the solutions that are being built for containers using containers in Open Source!\n\nDo contribute and earn your Open Source Karma by weighing in your thoughts at this *[Reddit thread (r/Docker)](https://www.reddit.com/r/docker/comments/6l0y3v/persistent_storage_with_docker_in_production/) or at [OpenEBS Slack](http://slack.openebs.io/) or at the [GitHub/Container-Storage-Interface](https://github.com/container-storage-interface/spec)*\n","notHasFeatureImage":true,"slug":"tech-enthusiasts-weighing-in-on-container-native-storage"},{"id":125,"title":"OpenEBS building Go Storage Kit Project — Maya","author":"Satyam Zode","author_info":"Go Developer @openebs | Open Source Contributor | Avid Learner","date":"25-07-2017","tags":["Golang","OpenEBS","DevOps","Container Orchestration"],"excerpt":"I attended GopherCon India 2017, there was a talk on “Package Oriented Design In Go” by William Kennedy. In that talk, William explained some really important and thoughtful design principles which we can apply in our day-to-day life, while writing Go.","content":"\n## Motivation\n\nI attended [GopherCon India](http://www.gophercon.in/) 2017, there was a [talk](https://youtu.be/spKM5CyBwJA?list=PLFjrjdmBd0CoclkJ_JdBET5fzz4u0SELZ) on “Package Oriented Design In Go” by [William Kennedy](https://twitter.com/goinggodotnet). In that talk, William explained some really important and thoughtful design principles which we can apply in our day-to-day life, while writing [Go](https://golang.org/project/). I have attempted to absorb some of the design philosophies I learned at GopherCon into practice at [OpenEBS](https://github.com/openebs). At [OpenEBS](https://github.com/openebs), as an open-source and growing Go project, We value Go principles and We try hard to leverage Go’s offerings.\n\nBriefly, OpenEBS is a [container-native storage](/blog/cloud-native-storage-vs-marketers-doing-cloud-washing) that is built from containers for enabling stateful containers to get into production. Being container-native, OpenEBS augments the Container Orchestration (CO) Layers like Kubernetes, DockerSwarm, Mesos, etc., with Storage Specific Orchestration capabilities using “Maya” — Magic! I contribute mainly towards the “Maya” — which is a set of containerized control plane applications for hooking into several modules like the configuration, monitoring, and alerting of CO.\n\n## What is the Go Kit Project?\n\nTo understand in plain terms, let us take an example where we end up writing the same Go packages, again and again, to do the same task at different levels in the different Go projects under the same organization. We are all familiar with the custom logger package in the different Go projects.\n\nWhat if, the custom logger package is the same across the organization and can be reused by simply importing it, then this custom logger package is the perfect fit for the Kit project. The advantages of this approach go beyond avoiding duplicate code, improved readability of the projects in an organization, to savings in terms of time and cost as well :-)\n\nIf you go through Bill’s talk, you will notice that the Kit project is characterized by Usability, Purpose, and Portability. In this blog, I will discuss how I have applied the refactored code to use the “Kit Project” pattern for maya.\n\n## How to convert existing projects to have “kit”\n\nOpenEBS being a container-native project is delivered via a set of containers. For instance, with OpenEBS 0.3 release we have the following active maya related projects:\n\n1. openebs/maya aka ****maya-cli**** : is the command-line interface like kubectl for interacting with maya services for performing storage operations.\n2. openebs/mayaserver : or ****m-apiserver**** abstracts a generic volume API that can be used to provision OpenEBS Disks using containers launched using the CO like K8s, nomad, etc.,\n3. openebs/****openebs-k8s-provisioner**** : is the K8s controller for dynamically creating OpenEBS PVs\n\nWith these projects, we are already seeing how code gets duplicated when each of these projects is independently developed. For example *maya-cli* and *openebs-k8s-provisioner* both need to interact with *maya-apiserver*, which resulted in maya-apiserver-client code being written in *maya-cli* and *openebs-k8s-provisioner*. Similarly, *openebs-k8s-provisioner* and *maya-apiserver* have duplicated code w.r.t to accessing the K8s services.\n\nTo avoid this duplicity of code using the kit project, we are transforming OpenEBS/maya into a Kit project for the Application projects like [maya-apiserver](https://github.com/openebs/mayaserver), openebs-k8s-provisioner, and many more coming up in the future. openebs/maya contains all the kubernetes & nomad APIs, common utilities, etc. needed for the development of maya-apiserver and maya-storage-bot. In the near future, we are trying to push our custom libraries to maya. So that, it will become a promising Go kit project for the OpenEBS community.\n\nLets now see, how maya (as a kit project) adheres to the package oriented design principles:\n\n- ****Usability**** \nWe moved common packages such as orchprovider, types, pkg to maya from maya-apiserver. These packages are very generic and can be used in most of the Go projects in the OpenEBS organization. Brief details about new packages in Maya.\n1. Orchprovider: orchprovider contains packages of different orchestrators such as Kubernetes and nomad.\n2. types: types provide all the generic types related to orchestrator.\n3. pkg: pkg contains packages like nethelper, util, etc.\n4. volumes: volumes contain packages related to volume provisioner and profiles.\n- ****Purpose**** \nWhile the Packages in the Kit project are categorized as per the functionality, the naming convention should ideally provide the reader with the information on what the package “provides”. So, the packages (in the kit project) must provide, not contain. In maya, we have packages like types, orchprovider, volumes, etc. name of these packages suggests the functionality provided by them.\n- ****Portability**** \nPortability is an important factor for packages in the kit project. Hence, we are making maya in such a way that it will be easy to import and use in any Go project. Packages in the Maya are not a single point of dependency and all the packages are independent of each other. For example, types directory contains versioned Kubernetes and Nomad packages. These packages are simply importable to any project to use Kubernetes and Nomad APIs.\n\n## Example usage of maya kit project\n\nMaya-apiserver uses maya as a Kit project. Maya-apiserver exposes OpenEBS operations in form of REST APIs. This allows multiple clients e.g. volume-related plugins to consume OpenEBS storage operations exposed by Maya API server. Maya-apiserver will use volume provisioner as well as orchestration provider modules from Maya. Maya-apiserver will always have HTTP endpoints to do OpenEBS operations.\n\nSimilarly, openebs-k8s-provisioner will use the maya-kit project Kubernetes API to query for details about the Storage Classes, etc.,\n\nAnother usage is of the maya-kit project, maya-apiserver client that is accessed by maya-cli as well as the openebs-k8s-provisioner to talk to maya-apiserver.\n\n## Conclusion\n\nGo Kit project should contain packages that are usable, purposeful, and portable. Go Kit projects will improve the efficiency of the organization at both human and code levels.\n","notHasFeatureImage":true,"slug":"openebs-building-go-storage-kit-project-—-maya"},{"id":126,"title":"Cloud Native storage vs. marketers doing Cloud Washing","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"18-07-2017","tags":["Cloud Native","DevOps","Docker","Kubernetes","OpenEBS"],"excerpt":"Let’s try to protect the phrases Cloud-Native storage and Container Native storage; or maybe we should start using a yet more specific phrase such as Containerized Storage for Containers","content":"\nSome years ago, back when I was founding CEO of Nexenta during our high growth days, I tried to bring some rigor to the discussion around “software-defined storage” by setting out a definition. My basic point was that it wasn’t enough for storage to have APIs of some sort and hence to be able to be controlled by software -> in addition the storage ought itself to *be software*, and hence be able to be provisioned as software and so forth.*1*\n\nI wrote those blogs setting out a definition for software-defined storage because I was disgusted by the number of legacy storage vendors that went from denying that software-defined storage was a thing to, almost overnight, claiming that they *already did* software-defined storage. The risk was that the large sales and marketing budgets of legacy vendors would drown out the innovative companies that were actually building software that delivered on the benefits of software-defined storage thanks to, you know, *actually being software-defined*.\n\nFast forward 6 or 7 years and — **here we go again!**\n\nThis time we are seeing “Cloud Native storage” as a term being abused along with “Container Native storage.” Logically Cloud Native storage should be storage software that itself is, you know, *Cloud Native*. And that term is defined by people far smarter than me, however typically it includes some sense of microservices, and [12-factor approaches](https://12factor.net/), and the ability to consume cloud services. One great and lengthy definition of the term is explained by the inestimable Joe Beda [here](https://blog.heptio.com/cloud-native-part-1-definition-716ed30e9193).\n\nWe are even seeing Cloud Native storage applied to *any storage that serves Cloud Native applications*. Thanks to the work of Docker and Kubernetes and others, nearly any storage *can* serve storage for containers — albeit in a way that typically requires the user to change the way that they run the containers that are attached to the underlying storage to such an extent that many of the benefits of containerization are lost; I talk more about benefits of truly container-native storage below. Calling hardware based storage that ties into a Kubernetes environment via plug-ins “Cloud Native storage” is absurd and yet it is starting to happen. Such storage is no more Cloud Native than any legacy monolithic application is “Cloud Native” just because you’ve been able to cram it onto a container somehow.\n\nSpeaking of cramming monolithic apps into containers, if you have a famously difficult to manage monolithic storage solution that you cram onto a container or two — even if you integrate it nicely into Kubernetes or other Cloud Native environments — what you have maybe useful however it is NOT Cloud Native. Because, you know, it predates 12 factor and cloud-native approaches to building software by many years and, as such, is no more Cloud Native than ye old SAP or other traditional n tier app.\n\nAhh, that feels better.\n\nBut you might ask — *so what*? What’s the big deal about marketers abusing the English language for their own ends and distorting the definition of Cloud Native storage?\n\nWell, that comes down to whether we believe that the benefits of Cloud Native applications could apply to building storage software in a cloud-native manner. Is what’s good for the goose, good for the gander? Is Cloud Native storage even a term worth fighting over? Could a truly Cloud Native storage solution deliver benefits much as other Cloud Native software does?\n\n![Picture showing gooses](/images/blog/goose.png)\n\n**Let’s all be cloud native like the whale: goose, gander, OpenEBS mules, and friends**\n\nWell, why wouldn’t benefits such as:\n\n- better resiliency\n- better resource utilization\n- innate horizontal scalability\n- ease of troubleshooting\n\nand other benefits that apply to decomposed microservice based applications (aka “Cloud Native”) apply to software that is delivering storage?\n\nPerhaps just as importantly, wouldn’t the teams and communities that are building the software be able to be better organized — themselves loosely coupled and able to *DevOps on* — than those that are working on monolithic software? [Conway’s Law ](https://en.wikipedia.org/wiki/Conway%27s_law)does seem to be a major reason for the success of Cloud Native and microservices.\n\nThe benefit of that better team organization in my experience can be massively greater developer productivity. As many have pointed out, the productivity of developers in high performing DevOps / Microservices organizations is far, far superior; these developers deliver usable code 10–100x more quickly than developers stuck building to long release cycles with massive dependencies.\n\nSo, in short, here is my request of you. Just as we have been so careful in (not entirely successfully) protecting the term DevOps, let’s try to protect the phrases Cloud Native storage and Container Native storage; or maybe we should start using a yet more specific phrase such as ***Containerized Storage for Containers*** (warning, that’s a term we coined at OpenEBS). Otherwise, we risk polluting everyone’s efforts for a better way to build and run software in a Cloud Native way with marketing BS.\n\n1. (note — as I’ve tried to [explain elsewhere](/blog/software-defined-storage-finally), much of what we hoped to achieve with software-defined storage is only now being enabled, largely thanks to Kubernetes and other open orchestrators and containers as well).\n","notHasFeatureImage":false,"slug":"cloud-native-storage-vs.-marketers-doing-cloud-washing"},{"id":127,"title":"Container Native Storage builds its High Availability in style","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"17-07-2017","tags":["Docker","High Availability","Kubernetes","OpenEBS","Storage"],"excerpt":"Infrastructure components are the toughest to build. It is meant to work always by default.","content":"\nInfrastructure components are the toughest to build. It is meant to work always by default. It must be telecom grade as scores of applications rely on the infra component’s zero downtime, no deterioration in QoS, etc. However, we all know software has got defects and it is no different for infrastructure software components as well. One of the things the software can do is implement high availability as its core feature and somehow buy time (by the virtue of HA) when one of these components experiences a breakdown due to defects or otherwise.\n\nImplementing the HA piece seems to be the elixir to all the unforeseen issues. Is that so simple then? Programmers who have built the HA in the past will swear to its complexity. It is another piece of software logic that is very difficult to get right the first time or even after a couple of major releases.\n\n> The non HA programmer can have the flexibility to err. Well, a programmer is supposed to be a human after all. However, this option is ruled out for the HA logic implementer.\n\nHaving said this, implementing high availability logic in a '*Container Native*' software solution has reasons to cheer. Most of the container orchestration platforms have abstracted the scheduling, placements, evacuations, prioritized jobs, and whatnot and thus have freed the average programmer from these overwhelming tasks.\n\nAll of a sudden, programming becomes fun again (*HA logic becomes more like placing the [lego](https://en.wikipedia.org/wiki/Lego) blocks in a fashion/pattern to get your favorite character into action*).\n\nProgrammers now need to build the required HA logic by tying together appropriate placement components that result in a customized HA solution. We at OpenEBS [***toolroom***](https://github.com/openebs/) have been trying to maximize the offerings provided by [*Kubernetes*](https://kubernetes.io/) towards achieving a true container-native solution for OpenEBS high availability. All the advancements in Kubernetes will make OpenEBS better and the scenarios handled by OpenEBS (*which will never be simple*) can become a point of reference in Kubernetes.\n\nLet’s get into some storage specific HA basics:\n\n1. A typical persistent storage solution would require its data to be replicated across hosts within a cluster. There are also cases, where the production scenarios demand this replication to be across geographies. In addition, hybrid cases require some replicas to reside within the cluster while other replicas to be placed across zones, regions, etc.\n2. Since storage solutions derive their power from their underlying hardware resources, it makes sense to earmark exclusive hardware for storage components. In other words, the host(s) would want to avoid software components that are not storage specific.\n3. Solutions around placements alone are not sufficient for storage software to be highly available. There can be cases where a storage controller does not support active-active mode. Now relying just on placements can not avoid application outages due to these storage hiccups (*i.e. storage protocol connection breaks resulting in breakdown of the applications consuming this storage*). The evacuation should be fast & its new placement cannot take any time longer. Appropriate policies should be in place to let these storage components be evacuated early before the application components. This becomes more essential during a node crash where every component will fight for survival.\n4. Storage HA cannot rely on half-baked evacuations that may lead to split-brain conditions. Storage should either be evacuated or be let to die along with the crash. There should be no cases of hangs, stalls, freezes, you name it. There can be only one option & that is:\n\n***100% successful evacuation as well as 100% successful re-scheduling.***\n\nIn some cases, 100% successful evacuation might imply the use of the logic called [**stonith**](https://en.wikipedia.org/wiki/STONITH) before boarding the re-schedule flight.\n\nWith the above HA basics (*you may like to call it hardships*) in our mind let us find if the most popular container orchestrator has any feature, policy, etc that can simplify HA in storage. While looking at Kubernetes (version 1.7) I could find a couple of policies that can help storage build its HA story with ease.\n\n> One: ‘Node affinity’ is a feature that constrains which nodes your pod is eligible to schedule on, based on labels on the node.\n\n> Two: ‘Inter-pod affinity and anti-affinity’ is a feature to constrain which nodes your pod is eligible to schedule on based on labels on pods that are already running on the node rather than based on labels on nodes.\n\n> Three: ‘Taints and tolerations’ work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.\n\nDoes the above sound familiar? It does not introduce any new jargon except for the term 'Pod' and perhaps 'Labels'. A pod is a logical concept for a container or bunch of containers while labels can be understood as tags. Now if our persistent storage is really a piece of software that runs from within a container all the properties that Kubernetes exposes can be applied against this storage software & let it construct (or extend or even plug) its 'High Availability' feature.\n","notHasFeatureImage":true,"slug":"container-native-storage-builds-its-high-availability-in-style"},{"id":128,"title":"Test Driven Development — The DevOps Way","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"13-07-2017","tags":["Docker","Kubernetes","OpenEBS","TDD","Software Development","DevOps"],"excerpt":"TDD is the abbreviated form for `Test Driven Development`. It might also be true in-case of newbies to have never heard of TDD in the current season where DevOps, NoOps, DataOps, ML, IOT rule the roost.","content":"\nWe might have all heard of TDD, don’t we? TDD is the abbreviated form for 'Test Driven Development'. It might also be true in case of newbies to have never heard of TDD in the current season where DevOps, NoOps, DataOps, ML, IOT rule the roost.\n\nTDD in simpler terms would be planning and implementing test logic before implementing its development logic. However, I find it very difficult to do this in practice. What I do instead is a little bit of planning & start building the development logic. In the next iteration, I would start writing corresponding test logic i.e. unit test programming (*remember those jUnit days*).\n\nReaders may still be wondering how “*DevOps is related to TDD*”. Well, here it is:\n\n> *How about using the best of DevOps tools towards TDD implementation?*\n\nThe logic behind this is no brainier.\n\n> I find the very best of the open-source community busy in building & tuning tools old & new that are meant to satisfy DevOps culture & practices.\n\nIt makes sense for us to use these tools that meet our TDD needs.\n\nThe question that might be lingering now would be about DevOps vis-a-vis TDD. ***Is TDD not part of DevOps culture***? The answer is an emphatic yes. However, I am trying to pick up one thread at-a-time from the DevOps culture/practice. That thread here is TDD & is the focus area for this article. I would like to emphasize that TDD does not necessarily mean unit testing. It can be thought of as applying test logic, scenarios, etc as the team continues with its development efforts, that in-turn brings in changes, discoveries, ideas in test as well as in development. ***To sum it up, it tries to remove the pain associated with latter stage design changes*** In addition, the team is ***no more afraid*** of new features, bug fixes, requirement changes, etc. due to the presence of test logic that gets triggered due to any of these changes.\n\nHope to talk again about some tangible implementations we are doing in [OpenEBS](https://openebs.io/blog). I would end the story here with a tip.\n\n> *The popular currency of current age DevOps’ tooling is Docker & Kubernetes.*\n","notHasFeatureImage":true,"slug":"test-driven-development-—-the-devops-way"},{"id":129,"title":"OpenEBS on the Growth Path! Releases 0.3","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"29-06-2017","tags":["Jupyter Notebook","Kubernetes","OpenEBS","Postgresql","Storage Containers"],"excerpt":"It gives me immense pleasure to watch this baby (OpenEBS) successfully cross another milestone with its 0.3 release.","content":"\nIt gives me immense pleasure to watch this baby (OpenEBS) successfully cross another milestone with its 0.3 release. OpenEBS 0.3 not only marks the delivery of additional functionality but also comes with an all-rounded growth.\n\nThanks to our growing community (as depicted in the diagram below), we have had great conversations with thought leaders in the container world, who are constantly looking for ways to simplify the usage of Containers and Container Orchestrators. We seem to have hit a sweet spot in terms of what we set out to deliver with OpenEBS, which is resonating loud and clear with anyone, who is venturing to run persistence workloads in containers.\n\n![OpenEBS community tracker](https://cdn-images-1.medium.com/max/800/1*wnMG__-zl8yO06f7AAJh5w.png) \n\nWhile we have seen steady growth in the followers/subscribers on our community channels, we have also seen steady growth in the contributors to the project and we are happy to announce that we have been able to expand our team members at CloudByte(from across the world), to work full-time on OpenEBS.\n\nSignificant functionality changes include:\n\n- OpenEBS Storage Containers (VSMs) are completely orchestrated and managed via Kubernetes.\n- VSMs seamlessly work with your preferred pod networking\n- OpenEBS supports dynamic provisioners like EBS and GCP\n- *maya*, the Storage CLI can be used to fetch granular IO usage statistics similar to your Amazon Cloud Watch\n\nAdditional changes can be looked up at our [OpenEBS Project Tracker](https://github.com/openebs/openebs/wiki/Project-Tracker).\n\nWe are very attentive towards making the onboarding experience as smooth and simple as possible. OpenEBS 0.3, fully integrates into Kubernetes, allowing you to use OpenEBS storage on your existing Kubernetes using the following simple commands:\n\n---\n\n`kubectl apply -f openebs-operator.yaml`\n`kubectl apply -f openebs-storageclasses.yaml`\n\n---\n\nThe YAML files can be downloaded from [here](https://github.com/openebs/openebs/tree/master/k8s). If you don’t have an existing Kubernetes cluster, you can easily set it up using our [vagrant box](/blog/multinode-kubernetes-16-cluster-provisioning-made-easy-using-sandbox-vagrant-box). Detailed instructions for running OpenEBS on GKE can be found [here](https://github.com/openebs/openebs/blob/master/k8s/hyperconverged/tutorial-configure-openebs-gke.md). You could also use our [Ansible Playbooks](https://github.com/openebs/openebs/blob/master/e2e/ansible/openebs-on-premise-deployment-guide.md) to setup Kubernetes cluster with OpenEBS storage on-premise on physical servers OR virtual machines.\n\nWe also have (with the help of the community) focused efforts in providing [examples](https://github.com/openebs/openebs/tree/master/k8s/demo) of persistence work-loads. In the current release, we are happy to provide the following:\n\n- Clustered PostgreSQL Setup (inspired from [CrunchyData](https://www.crunchydata.com/))\n- Containerized Jupyter Notebooks (inspired from [kensu.io](http://www.kensu.io/))\n\nThe growth, in terms of the distributed teams, comes with the challenges of collaboration and maintaining the quality of the commits. One of our focuses with OpenEBS release 0.3 has been to augment our repositories with the tools that keep a watch on the code and product quality — we are now fully integrated with Travis, Codecov, GoReport and have built Ansible Playbooks for running CI via Jenkins.\n\n---\n\nAs we accelerate towards our enterprise version of the product towards the end of this year, we are all excited about the immediate milestones w.r.t:\n\n- Flush-out the details of Storage Infrastructure as Code (single source of truth)\n- Integrating into Container Monitoring Frameworks\n- Providing a UI\n- Optimize for high performance storage and networking infra.\n- Provide additional examples of Persistence Workloads that make use of the Application aware intelligence provided by OpenEBS\n\nWe always love to hear from you and what you think are the biggest storage operations pain points that you want to see eliminated. Do drop into our *[Slack Channel](http://slack.openebs.io) or stay connected with us via our other channels [Twitter](https://twitter.com/openebs), [GitHub](https://github.com/openebs/openebs/)*.\n","notHasFeatureImage":true,"slug":"openebs-on-the-growth-path!-releases-0.3"},{"id":130,"title":"Storage infrastructure as code using OpenEBS","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"29-06-2017","tags":["Containerised Storage","DevOps","Kubernetes","OpenEBS","Stateful Applications"],"excerpt":"With a vision to become de-facto block storage choice for stateful container applications, OpenEBS brings out many unique features to the open source storage space.","content":"\nWith a vision to become de-facto block storage choice for stateful container applications, OpenEBS brings out many unique features to the open-source storage space. One of the unique features that I talk about in more depth in this blog is “how storage platform builders and storage platform users can manifest their requirements as code using OpenEBS”.\n\nWith all the agility in Container and DevOps space, we are moving towards the thinking — “using infrastructure components such as network and storage for the application development should not be external to application development”. What I mean by this is, just like you write the code for your applications, you should be able to write some code or configuration for obtaining the storage and network infrastructure and you should get it.\n\n#### The YAML: Infrastructure coding language\n\nIn the paradigm of infrastructure as code, YAML is becoming the default coding language. Thanks to the increasing usage of YAML config files by the DevOps eco-system, they are becoming the default developer language to write the infrastructure code for the application development.\n\nSo, let us see what are the areas in which storage as infrastructure has to be thought in terms of code? This can be answered using the angle of who interacts with storage or the user persona. The simple answer would be the application developer and storage platform engineer. Next, what are the challenges today for a developer of a stateful application and the developer or engineer of an underlying persistent storage platform? In short, the developer needs\n\n- Some amount of persistent storage\n- With guaranteed availability\n- With guaranteed latency\n\nThe developer needs to quantify the amount of storage and the latency levels, which is a challenge always. The developer always starts with a guesstimate and refines later. The guaranteed availability is a characteristic of the underlying storage platform, but the developer may still need to decide some aspects related to the availability such as “how many copies of the data is needed, how often backups are to be taken, etc”. All these decisions to be taken by the developer, bring in the need for storage expertise or knowledge to the application developer, which usually is an additional burden to the developer. Traditionally, these decisions are taken by storage administrators who are adjacent to the application developers, creating a gap between what is needed and what is available. As the application scales, this gap widens, the application fails. **Yes, because something went wrong with storage!!**\n\nThe solution to this well known and well understood problem is to move towards a system where the application developer auto-magically gets help from the underlying storage system and he/she can continue to express/code the storage needs through the YAML language. The “so-called” storage administrator in the enterprise is going to be helped with an intelligent-storage-robot working continuously to align the storage to the application needs. With OpenEBS, the “intelligent-storage-robot” comes as an inherent component of the storage platform. The storage administrator can scale the platform and manage it at ease and do the job very well.\n\nThe intelligent-storage-robot or what we call OpenEBS as “Storage bot” handles the tough job of automating the storage actions such as\n\n- Updating the storage catalogues for application developers\n- Work with the storage platform to identify the capacity change needs, migration needs, backup needs and coordinate with the application\n- Work with the application for scheduling application aware snapshots, capacity management, data migration changes, automated data recovery etc\n- Work with a centralized ML engine (if there is one) to upload the application behaviour or to learn about the application behaviour so that it becomes more intelligent\n\n![Storage bot helping app developer of stateful container application and storage admin of the platform](https://cdn-images-1.medium.com/max/800/1*3wWTPR7i1gAVagzBYlOBmg.png)\n(***Storage bot helping app developer of stateful container application and storage admin of the platform***)\n\nAs depicted above, the storage solution designer is replaced by a storage-Bot (code) which is more effective as it predicts and updates the parameters of the storage platform as well as the application in real-time.\n\n#### OpenEBS helps to manage storage needs with YAML language/code\n\nOpenEBS divides the storage configuration into two parts. One, that the storage platform engineer has to design/manage and the other that the application developer has to design/manage.\n\n**The platform YAML file**\n\nThe platform configuration is specific to OpenEBS and the initial version is available at -\n\n[https://github.com/openebs/openebs/blob/master/k8s/openebs-config.yaml](https://github.com/openebs/openebs/blob/master/k8s/openebs-config.yaml)\n\nThe storage platform engineer develops this YAML file to scale the storage platform from few nodes to thousands of nodes across multiple locations. The YAML files can be version-controlled in a git repository and deployed into production to roll out the storage build-up process.\n\n![Storage platform deployment through YAML file automation](https://cdn-images-1.medium.com/max/800/1*WiZS5A4iLPeMtwxUmdxunQ.png)\n(***Storage platform deployment through YAML file automation***)\n\nA real example of this automation would be something like —\n\n---\n\n*the storage bot senses that there is a flood of large block size reads about to hit on a postgres-volume-246 which has its copies on node2 and node4 and to satisfy the latency needs of this volume, the read-cache on node4 is insufficient and one of the spare SSD on node4 can be re-purposed as read-cache temporarily. Storage bot learns this situation, makes the decision to do so and modifies the openebs-config.yaml on node4 as a new git version and rolls out the config file. Once the workloads are processed successfully after a few hours/days, the read-cache SSD is re-purposed back by the storage-bot either by updating the config again or by rolling back the config change.*\n\n---\n\nThe above is a very simple example and more complex things like “automated data migration to a correct node or correct tier, adding compute and memory resources to the storage controller nodes on the fly, automated software upgrades, etc” can all be achieved by the storage-bot using the git-versioned openebs-config.yaml files\n\n**The application YAML file**\n\nNow to the application developers paradise 🙂\n\nOpenEBS readily provides the storage configuration templates/storage catalogs for various stateful applications and application variants. The storage classes will be an ever-growing list with more participation from the community, but the initial version is at — [OpenEBS Storage Classes](https://github.com/openebs/openebs/blob/master/k8s/openebs-storageclasses.yaml)\n\nThe storage class will help the application developer with various storage config parameters. Some of the possibilities are:\n\n— Initial capacity, capacity growth rate per week/month, latency expectations, active data size (hot/cold data ratio), backup schedules, application data patterns, number of copies to be saved, typical instructions during upgrades, data migration boundary conditions, etc\n\nAs you can see, the above actions form a major part of the job of storage administrator who plans, designs, and executes the storage parameters. In the case of OpenEBS, the application developer can code these requirements into the application YAML file and the OpenEBS storage bot will enforce and help continuously optimize this file as well. This, I believe is a huge step forward in the journey of simplifying the storage infrastructure interface to the application developers.\n\nOpenEBS is in the early days. With the release of OpenEBS 0.3, the DevOps users have an option to get the container-native block storage for their stateful workloads orchestrated by Kubernetes. OpenEBS 0.3 release is the first step in making the storage infrastructure as code possible.\n\nLooking forward to receiving feedback/comments/criticism on this topic. We are lurking on slack at [https://openebs-community.slack.com/](https://openebs-community.slack.com/)\n","notHasFeatureImage":true,"slug":"storage-infrastructure-as-code-using-openebs"},{"id":131,"title":"Storage is Dead! Long live OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"28-06-2017","tags":["DevOps","Kubernetes"],"excerpt":"In this blog I share something I’ve been saying for a while now that people suggested might just be audacious enough to spark a conversation.","content":"\nIn this blog, I share something I’ve been saying for a while now that people suggested might just be audacious enough to spark a conversation.\n\nPlus — I explain why I’ve decided to help found another storage project — [OpenEBS](http://www.openebs.io/) — and to become CEO of the underlying company, [CloudByte](http://www.cloudbyte.com/), at a time in which venture capital funding to space has declined (at least if you accept Rubrik and [Cloudian](https://cloudian.com/) and similar) funding as outliers :)).\n\nTL;DR: the old storage industry is ****dying**** like many stagnant and declining IT sectors before it, killed through outdated business practices and architectures and the emergence of a new set of personas that have radically different requirements, expectations, and even tastes. The new architectures and business models already exist — though they are unevenly distributed.\n\nI lay the story out in three points. I keep it brief and link to underlying technical resources w/ code and demos via those links for those that are interested. Please discuss below and/or at: [@epowell101](http://twitter.com/epowell101). Better yet, click here to be invited to our Slack channel: [https://openebsslacksignup.herokuapp.com](https://openebsslacksignup.herokuapp.com/)\n\n****Storage is dead? How so?****\n\n*On the one hand, massive data growth continues.* It is easy to look at a graph like the one above showing ongoing exponential growth in data created and conclude that the funding winter that has hit storage is just another irrationality of the venture market. Perhaps it is like 2006–2008 when too many of the top enterprise technology VCs had a couple of walking dead storage companies in their portfolio and wouldn’t or couldn’t take on new investments.\n\nAnd many thanks to Mary Meeker and Kleiner Perkins for her [annual report](http://www.kpcb.com/internet-trends) on trends for the graph.\n\n****Cloud****\n\n*On the other hand, cloud.* The major cloud vendors are a) truly major and b) see storage as a way to control customers.\n\nTo the ‘truly major’ point — the three largest cloud providers are some of the largest companies the world has ever seen. Those of us that remember the power of WinTel in the 90s and early 2000s need to remember that this time is different in part because the companies are massively more wealthy thanks in large part to having business models that transcend technology.\n\nTo the ‘control customers’ point — data gravity exists. And it is further leveraged by cloud providers making it free to transfer data to them and rather expensive to get it back.\n\nWhich brings us to DevOps and cross-cloud services…\n\n****DevOps and cross-cloud services****\n\nAs I learned from my experience as a co-founder and CEO of [StackStorm](https://github.com/StackStorm/st2), DevOps is a radically better way of building and running software — as well as being a cultural movement.\n\nAnd while there are many, many characteristics of what constitutes a “DevOps environment” — one important aspect as I see it is that the intent of the developer can be fully manifested by the underlying infrastructure and related services.\n\nAnd a second aspect is that many capabilities are consumed by the applications and the application developers as services. This fits with the use of microservices as well, of course.\n\nLast but not least, DevOps is a culture and a set of practices that, as demonstrated by the story of the crucial book the Phoenix Project, is motivated in part by a realization of how broken, slow, dysfunctional, unfair, and brittle, proprietary IT systems and IT operations traditionally have been. From this perspective, applying infrastructure as code in part to ensure cross team transparency and blameless postmortems and so forth are not just examples of useful techniques — they are manifestations of the values of a better culture.\n\nOne strength of clouds like AWS is that they enable you to consume storage in a DevOps friendly way — your orchestration or other automation can deal with the provisioning and management (or can rely on the solutions provided by the cloud provider) so that intent can flow from the application through the infrastructure; in addition, the infrastructure is there to be consumed dynamically, via APIs. Unfortunately, the cloud providers themselves have become the new proprietary, which makes some in the DevOps world a bit uncomfortable as in the past lock-in was used by proprietary vendors to maximize their own prices at the expense of their customers. Kubernetes especially offers a chance to keep the wiring free from the cloud providers, hence providing some hope of avoiding lock-in.\n\n****Uhh, huh — and OpenEBS?****\n\n![Uhh, huh — and OpenEBS?](/images/blog/uhh-huh-openebs.png)\n(***Uhh huh — data, cloud, DevOps, lock-in and, err, OpenEBS?***)\n\n\nIn short, the highest level vision for OpenEBS is one in which the DevOps friendly aspects of running storage in the cloud is now available on-premise and across your clouds, thus freeing you from vendor lock-in much in the way that Kubernetes helps you avoid the risk of locking yourself in through the use of cloud service specific wiring and workflow.\n\nWhat if your developers and DevOps teams could orchestrate your storage controllers just like other containers? How much more productive could they be?\n\nAnd what if that set of storage services allowed you to treat your stateful workloads on containers much like the ephemeral containers that have proliferated?\n\nAnd what if your storage and CIO teams were able to establish policies that governed your data without impeding the agility that was your purpose in moving towards containers and DevOps in the first place?\n\nLast but not least, what if, having tooled your environment to work with Amazon it just worked when you added the on-premise and cross-cloud OpenEBS? Wouldn’t this limit your lock-in with any one cloud vendor?\n\nOpenEBS remains early. Today we are at `release 0.3` and are just starting to be used by enterprises like Cap Gemini and others to deliver storage for stateful workloads on containers.\n\nI’m back into the storage industry because I’m sure that OpenEBS has a good chance to allow storage to slip into the background and “just work” to support the incredible boosts in agility delivered by DevOps.\n\nA bunch of us imagined a better storage industry — one more software-like, one that eliminated vendor lock-in — back in the 2008 / 2009 time frame. Instead, we got some kick-ass flash and quite cool hyper-converged solutions PLUS, mostly, AWS just working (well enough) with a far better business model and ability to scale.\n\nSo vendor lock-in remained a problem. And storage far too often remains the bottleneck that slows down the good stuff, including the shift of real workloads onto containers.\n\n[OpenEBS](http://www.openebs.io/) is going to fix all of that and make storage and related storage services something you use to free yourself from lock-in while boosting your operational agility. Game on!\n\nYou can learn more about today’s release of OpenEBS 0.3 from our COO and co-founder’s blog [here](https://openebs.io/blog/author/uma-mukkara). I think you’ll agree, what’s possible now reinvents storage fundamentally — with your help we’ll turn this project into a company and a complete solution that we can all rely on.\n\nOther resources include demos of OpenEBS working seamlessly with Kubernetes and interesting stateful workloads including PostgreSQL and the Spark notebook Jupyter in our demo repo [here](https://github.com/openebs/openebs/tree/master/k8s/demo/crunchy-postgres).\n","notHasFeatureImage":false,"slug":"storage-is-dead!-long-live-openebs"},{"id":132,"title":"Data Scientists adopting tools and solutions that allow them to focus more on Data Science and less…","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"24-05-2017","tags":["Adalog","Data Science","Kubernetes","OpenEBS","Jupyter Notebook","Solutions"],"excerpt":"Data Science as we all know is becoming crucial to many if not most businesses around the world. Data Science and ML are decidedly, the most trendy skills that a lot of people are aspiring to acquire or be associated with.","content":"\nData Science as we all know is becoming crucial to many if not most businesses around the world. Data Science and ML are decidedly, the most trendy skills that a lot of people are aspiring to acquire or be associated with. In my recent interactions (in the context of hiring for OpenEBS) with college graduates and software engineers, the one question that pops up is, if they join OpenEBS, will they get a chance to work on ML / Data Science (and the answer is a definite “maybe” :)).\n\nThe lure towards “Data Science” has gone beyond the $$ paid to the Data Scientists. It is about the process of creating wonders. “Data Science” puts forth a unique mix of interesting challenges, that requires grappling with the enormous amounts of data and the mind numbing application of machine learning and computational algorithms, visualization, and soft skills. These aspects associated with “Data Science” are evolving, as depicted in this [data science Venn diagram](https://www.infoq.com/articles/christine-doig-data-science-team-discipline).\n\n![Data Science Venn Diagrams](https://cdn-images-1.medium.com/max/800/0*FEPtZ-3YF48YMkRs.)\n\nYou can see this trend expressed in this graph of searches on Google as well. An interesting point is that “Machine Learning” has been neck and neck with “Data Science” until recently when “Machine Learning” started to be searched approximately 30% more frequently. And the gap between searches about Data Science and about Data Scientists has widened in the last couple of years; it seems people are about 3x more likely to be interested in Data Science than in the Data Scientists that actually do the work 🙂\n\n![Graph of Machine Learning, Data Science, Data Scientist, and Deep Learning searches on Google](https://cdn-images-1.medium.com/max/800/0*Usx-pyQi3bHCTyJe.)\n\nInterestingly, we have seen early OpenEBS users and contributors asking about OpenEBS delivering storage to containers running distributing nodes for Kafka or Spark or other pieces of a Data Science pipeline. We wanted to learn more about the needs of these users.\n\nThankfully, we are a little bit of a sister company to [Kensu.io](http://www.kensu.io/) as our Chairman [Evan Powell](https://twitter.com/epowell101) has been advising [Kensu](https://twitter.com/kensuio) as well. So we got in touch with the Kensu.io founders [Andy](https://twitter.com/noootsab) and [Xavier](https://twitter.com/xtordoir), who are co-creators of the [spark-notebook](http://spark-notebook.io/), which is the leading Scala notebook for data scientists working in the Spark community. [Andy](https://twitter.com/noootsab) and [Xavier](https://twitter.com/xtordoir) and their team have built and have helped to run some of the largest Data Science pipelines in finance and related fields in Europe; plus they are O’Reilly authors. They have been really generous with their time.\n\nThe TL;DR is — *data scientists still spend way too much time hacking together Data Science pipelines and if they only could orchestrate the end to end pipeline with Kubernetes/Mesos (which seems to be common in these environments) so that storage itself worked simply, this could be more productive, could recreate their results more easily, and might even work together better.*\n\nIn the rest of the blog I set out a little more specifically what we have learned and we share the beginnings of a recipe for Data Science on containers with OpenEBS as the storage. Please provide feedback to this blog or join the now hundreds of people on the [OpenEBS slack channel](http://slack.openebs.io/) and post your comments and feedback there.\n\n**Data Science workflow hindered by infrastructure management tasks**\n\nThe workflow of a data science project typically comprises four stages as outlined below, with data scientists going back and forth between these stages, before arriving at product/insights in the form of a report, dashboard, or data service.\n\n![Data Science workflow](https://cdn-images-1.medium.com/max/800/0*UCJHD8p2bNOemyC9.)\n\nEach of these stages involves tasks that can be further classified into three distinct types depending on the nature of the task as follows:\n\nType 1: Data Science\n\nType 2: Best Practices or Governance\n\nType 3: Administer Infrastructure/Tools\n\nData Scientists are primarily interested in *Type 1: Data Science tasks*, some sample tasks under this category (at different stages of the workflow) are:\n\n![Data Science tasks at different stages of the workflow](https://cdn-images-1.medium.com/max/800/0*t41w0qHFapFBWtX_.)\n\n*One of the crucial requirements for the data scientist is to produce accurate and reproducible analysis by collaborating with larger teams. *[Martin Hack](https://twitter.com/mhackster)*, commented that he has come across situations where the accuracy of the models change from the time the models were generated to the time that they are presented to a wider audience. This is usually attributed to the change in the data using which the model was generated.*\n\nThe efficiency and confidence with which these tasks are performed or improved (in an iterative way) are dependent, in the current scenario, on some *best practices*. The best practices range from following certain naming conventions for the data files used to tuning infrastructure/tools — like databases, distributed systems that are used to run the analytical or modeling tasks.\n\nSome sample Type 2: Tasks at different phases of the Data Science workflow are as follows:\n\n![Tasks at different phases of the Data Science workflow](https://cdn-images-1.medium.com/max/800/0*IKPVzSqbyirvwZW5.)\n\nThe tasks mentioned above just provide a glimpse into the challenges associated with a typical data science workflow. In workflows that require creating models by using data from multiple live sources, tracking the accuracy of the data without having something like application error logs, becomes a harder problem to solve, if not impossible. This also leads to anxious data scientists who tend to keep their work private, unless a higher level of confidence is achieved on the models generated.\n\nTalking of confidence, today we take pride in writing code that goes into production on day one, using automated continuous integration systems. Kensu is doing something similar, which can be considered as *Continuous Integration mixed with Live Monitoring of Data Sources.*\n\nAndy and Xavier and the team are rolling out ***Adalog***, their software and SaaS solution, first for their existing professional service customers and then more broadly. While you can learn much more about Adalog on their website — [http://www.kensu.io](http://www.kensu.io) — as I understand its Adalog picks up all the relevant metadata of the data science pipeline and uses it, as well as the underlying systems themselves, to deliver compliance reporting and controls.\n\nData Science Notebooks (like [*Jupyter*](http://jupyter.org/), [Spark-Notebook](https://github.com/spark-notebook/spark-notebook), etc.,) and integrated products like Adalog have enabled improved and reproducible data since workflows, and deploying these notebooks towards containers have helped in boosting the productivity.\n\nYet, there are a few infrastructure related tasks that are not fully integrated into this workflow, which creates a manual dependency on the expertise of infrastructure administrators. At the same time, it also burdens the administrators/operations team to manage a new kind of workload in their environment.\n\nSome of the tasks that fall under this *Type 3: Infrastructure / Operations* tasks from (from a storage operations perspective) are:\n\n![Infrastructure / Operations tasks from (from a storage operations perspective)](https://cdn-images-1.medium.com/max/800/0*gcVdcir6RcbetPXe.)\n\n*Andy and Xavier stated that they have the integration pieces with the data science tools, he finds that storage infrastructure related challenges remain and many of these would seem to be best solved with tighter integration by-products like OpenEBS that are fully containerized and hyper coverged, with products like Adalog.*\n\n*Without a solution like OpenEBS that natively integrates into the orchestration and that containerizes the storage itself*\n\n- *Capacity management is very tricky, where some of the tasks involve non-deterministic data bloating. Hitting disk space errors may result in the loss of many hours of processing.*\n- *Managing the capacity for the local copies that need to be maintained for external data sources and keeping them in sync and distributing to team members can be very time consuming*\n- *The dynamic nature of jobs that require bursts of IOPS/Throughput means that to be safe the systems are massively over provisioned which then means you need a large budget request even to just mess around in trying out new models for example*\n- *Maintaining the provenance of the data along with the models for reproducing the results with the same accuracy they had during model generation is extremely difficult and without that the human workflow of many data science users is broken; imagine, for example, trying to compare the performance of multiple teams who are being evaluated based on the quality of their models if you cannot be sure that the training sets have remained trusted.*\n\n*Andy says, these infrastructure and hand-holding of the workflow operations are what makes Data Science more of engineering than science.*\n\n**Data Science workflow — storage management tasks automated by OpenEBS**\n\nDocker and Kubernetes have eased the operations around maintaining large scale computational clusters, and with containerized and hyper-converged storage like OpenEBS, the operations can now provide storage for millions of containers without having to maintain separate storage silos.\n\nKubernetes and OpenEBS will help in further simplifying the infrastructure management tasks for the data science workflows by supporting automated storage capacity management based on the data source changes, auto-tuning of the QoS depending on the priority of the analytical task, and provide data provenance without laying an extra burden on the data scientists by deeper integration with the notebooks.\n\n*For instance, the following is a glimpse of what can be achieved through deeper integration of Adalog with Kubernetes and OpenEBS:*\n\n1. *Data scientists starts a new notebook ( or project ). This project is instantiated on the container cluster (say Kubernetes).*\n2. *When a data source is specified for download, it would trigger provisioning of a new OpenEBS storage volume from this cluster for the data source. The capacity for this new volume is auto-managed by OpenEBS. Adalog would maintain the information of the volumes associated with a notebook.*\n3. *When the notebook finishes downloading the data, it will trigger to create a snapshot of the downloaded data. Notebook can allow for regular syncing with the source to check for updates and download the incremental data etc., Adalog would track the snapshots of the volume used by the notebook.*\n4. *As data scientists start exploring the data, there are provided with read-only access to the snapshot of the data-source that is already downloaded. And new storage volumes are created for storing the temporary files. Adalog can associate the volume created with the user who initiated the data generation.*\n5. *When a certain exploration is aborted or cleared from the notebook, the associated volume ( and all the temporary files) are also cleared from the storage. Adalog would keep track of the functions and models generated via the exploration phase along with metadata information about the results generated, even though the volumes (data) is deleted.*\n6. *OpenEBS volumes come with fine-grained allocation of resources and auto-tuning which can help with running high-priority modeling jobs faster than the others.*\n7. *From Adalog, the user can trigger OpenEBS volumes (data) to be backed up or restored from S3 bucket.*\n\nAs a next step, we are planning to provide a sandboxed version of adalog running on Kubernetes and storage managed by OpenEBS. Much like the vagrant box we built to enable OpenEBS to be tried out on Kubernetes, and then became a favorite tool of some largely for trying out Kubernetes — we are hopeful that this sandbox will be useful as well as being a way to see OpenEBS in action.\n\n*If you want to earn massive open-source karma (and stickers of course) — let us know on our *[*OpenEBS Slack channel*](http://slack.openebs.io/)* that you’d like to help with this solution building exercise.*\n","notHasFeatureImage":true,"slug":"data-scientists-adopting-tools-and-solutions-that-allow-them-to-focus-more-on-data-science-and-less…"},{"id":133,"title":"Multi-Node Kubernetes 1.6 Cluster provisioning made easy using SandBox (Vagrant box)","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"23-05-2017","tags":["Kubeadm","Kubernetes","OpenEBS","Vagrant","Virtualbox"],"excerpt":"Working on OpenEBS, a containerized storage for containers which is orchestrated by Kubernetes, most of our tasks, be it development, testing and demo require us to setup and modify the nodes in Kubernetes cluster.","content":"\n### Background\n\nWorking on OpenEBS, a containerized storage for containers which is orchestrated by Kubernetes, most of our tasks, be it development, testing and demo require us to setup and modify the nodes in Kubernetes cluster. In addition, a multi-node cluster is a must as we go beyond the initial development and testing, to explore the high availability, scale, performance, and upgrade aspects.\n\nWhile *minikube* and *minishift* provide an easy way to setup Kubernetes single node cluster — for multi-node cluster the fastest ways to get going are usually cloud or hosted solutions. ***kubeadm*** is the closest we can get to easily setup a cluster, but since it is still in alpha, we keep running into some issues like — [kubadm init v1.6.1 fails](https://github.com/kubernetes/kubeadm/issues/226)\n\n*An ideal solution for a developer would be a Kubernetes Sandbox. A sandbox that can be easily setup on a laptop and can work on the move (without net connectivity.) This Sandbox should be shielded from the different API or CLI changes that happen with the frequent releases of kubeadm and Kubernetes.*\n\nWe have used Vagrant, VirtualBox, and Atlas to do just that.\n\n![Vagrant, VirtualBox and Atlas](https://cdn-images-1.medium.com/max/800/1*7kkviZOwgh8ePDYRjFX0mQ.png)\n\n## Try It! It is Easy and Quick!\n\nOnce you have Vagrant (1.9.1 or higher) and VirtualBox (5.1.14 or higher) installed on your laptop/machine, just do the following:\n\nStep 1: Download the Vagrantfile from [OpenEBS Github](https://raw.githubusercontent.com/openebs/openebs/master/k8s/lib/vagrant/test/k8s/1.6/Vagrantfile)\nStep 2: Run **vagrant up**\n\nDetailed instructions can be found [here](https://github.com/openebs/openebs/tree/master/k8s/lib/vagrant/test/k8s/1.6).\n\nThe above two steps will provision the following:\n\n- Ubuntu VM with Kubernetes Master (kubemaster-01)\n- Ubuntu VM with Kubernetes Minion (kubeminion-01) associated with (kubemaster-01)\n- Setup *weave* as pod network\n- Setup the kubectl credentials ( admin.conf) on kubemaster-01\n- Sample Kubernetes pod YAML files are located on (kubemaster-01) under the directory (*~/demo/k8s/spec/*)\n\nIn addition to the above, the following OpenEBS provisioning tasks are also performed.\n\n- Install OpenEBS iSCSI FlexVolume Driver on the kubeminion-01\n- Ubuntu VMs installed with OpenEBS Maya Master and OpenEBS Storage Hosts. (If you don’t want to use the storage, you can skip the installation of these VMs. Check the customization steps below).\n\n## Customizing the Kubernetes Sandbox\n\nThe above instructions include setting up Kubernetes and OpenEBS as well, but you can easily customize the Vagrantfile to skip installation of OpenEBS by prefixing ENV variables before the vagrant command as follows:\n\n MM_NODES=0 MH_NODES=0 vagrant up\n\nSome of the configuration options available are:\n\n- KM_CPU — Number of CPUs for minion (*default 2*)\n- KM_MEM — Size of the RAM (in bytes) for minion (*default 2048*)\n- KH_NODES — Number of Kubernetes Minion VMs (*default 1*)\n- KH_CPU — Number of CPUs for minion (*default 2*)\n- KH_MEM— Size of the RAM (in bytes) for minion (*default 1024*)\n- MM_NODES — Number of OpenEBS Maya Master VMs (*default 1*)\n- MH_NODES — Number of OpenEBS Storage Host VMs (*default 2*)\n\nIf you are looking for an older release of Kubernetes, checkout — [kubernetes vagrant boxes with 1.5.5](https://blog.openebs.io/setting-up-kubernetes-1-5-5-cluster-with-vagrant-dda11e33b5bc)\n\n## Contributing to creating Kubernetes Sandboxes\n\nBtw, the process of creating these Kubernetes Sandboxes is Open Sourced.\n\nThe majority of the issues that are encountered during the Kubernetes cluster setup using kubeadm are related to the software api/cli options changed across different versions of kubeadm or the interfaces between kubeadm and Kubernetes. Another nagging issue is the need to have connectivity to the network.\n\nThese issues are resolved by having Sandbox (vagrant boxes) that pre-package the required software with versions that are compatible. The task of downloading the required software is automated via the scripts.\n\nOnce a VM is initialized with network/IP address details, certain initialization tasks will have to be executed. These are placed in the configuration scripts (which are also pre-packaged with the sandboxes) and are invoked from the Vagrantfile itself.\n\nCurrently, the Sandboxes use *weave* as a pod network, you can easily extend this to use a different scheme for pod network.\n\nIf you like to contribute or learn more about these box generation scripts, check out our [GitHub](https://github.com/openebs/openebs/tree/master/k8s/lib/vagrant) or join our [*Slack Channel*](http://slack.openebs.io).\n","notHasFeatureImage":true,"slug":"multi-node-kubernetes-1.6-cluster-provisioning-made-easy-using-sandbox-(vagrant-box)"},{"id":134,"title":"Rancher’s Longhorn announcement solidifies the OpenEBS story","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"18-04-2017","tags":["Docker","OpenEBS","Rancher","Stateful Applications","Longhorn"],"excerpt":"Today, Sheng Liang unveiled project Longhorn as a new way to build distributed block storage for cloud and container-based platform.","content":"\nToday, Sheng Liang [unveiled project Longhorn](http://rancher.com/microservices-block-storage/) as a new way to build distributed block storage for cloud and container-based platform. We, the OpenEBS team, are thrilled with this news as it solidifies our decision of using Longhorn as the underlying block storage system for OpenEBS.\n\nWhen I reviewed the OpenEBS vision with Sheng late last year, he introduced me to Longhorn project and asked me to study it and use it if it makes sense for OpenEBS. Clearly, Longhorn was sharing the OpenEBS block storage vision and we soon launched OpenEBS with Longhorn underneath.\n\nWhat I liked most about Longhorn was the nice separation of block protocol stack (Longhorn controller) and the actual block storage (Longhorn replica). This helped us containerize the storage software completely and efficiently. An OpenEBS VSM, or a storage pod, will have containerized longhorn controller and replica(s).\n\nAnother cool thing about Longhorn implementation is that it’s controller sits closer to the application docker containers thus enabling hyper-converged mode of deployment and it’s replica sits closer to the underlying storage with a simple sparse file based system for block data management.\n\nIt is so nice of Sheng to mention OpenEBS’s usage of LongHorn. We will continue to contribute to the performance tuning of LongHorn, improvements to S3 integration, rebuilding logic of replicas, etc\n\n### OpenEBS is an extension to the idea of Longhorn\n\nSheng Liang, in his blog, talks about Longhorn and other storage systems.\n\n“*We wrote Longhorn as an experiment to build distributed block storage using containers and microservices. Longhorn is not designed to compete with or replace existing storage software and storage systems…”*\n\nAs believers in the idea for which Longhorn is written, we are building OpenEBS as credible enterprise storage with Longhorn at it’s core. As part of the journey with CloudByte ElastiStor, where we have containerized the storage volumes a few years ago, we picked up tremendous amount of real world experience in delivering enterprise storage to the customers. All this experience is being put into work at OpenEBS. OpenEBS will have most of the enterprise features that Sheng mentions, if not more. Some of them will be “scalable and reliable distributed file systems, a unified storage experience, enterprise data management, crash consistency, near disk performance, etc”.\n\nAgain, a ton of thanks to the awesome folks who contributed to the development of Longhorn. With this announcement from Rancher, we are hoping to see more usage of Longhorn and OpenEBS. We are all ears to the community for any suggestions on OpenEBS and its usage of Longhorn ([slack.openebs.io](http://slack.openebs.io)).\n\nLong live Longhorn!!\n","notHasFeatureImage":false,"slug":"rancher’s-longhorn-announcement-solidifies-the-openebs-story"},{"id":135,"title":"OpenEBS sprinting ahead, 0.2 released","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"10-04-2017","tags":["Containerized Storage","Docker","Kubernetes","Microservices","OpenEBS"],"excerpt":"I am delighted that we have been able to push the OpenEBS 0.2 release for community consumption, a version that demonstrates that storage controllers can be containerized.","content":"\nI am delighted that we have been able to push the [OpenEBS 0.2 release](https://github.com/openebs/openebs/releases/tag/v0.2) for community consumption, a version that demonstrates that storage controllers can be containerized. This release comes with k8s flex volume driver called “openebs-iscsi”\n\nThe past week has been very exciting for the OpenEBS Team — engaging with container technology evangelists, enthusiasts, and users at the [Bangalore Container Conference — BCC 2017](http://www.containerconf.in/), followed by a very interactive meetup on the [Containerized Storage for Containers](http://www.containerconf.in/) with [Evan Powell](https://twitter.com/epowell101), [Ian Lewis](https://twitter.com/IanMLewis), and my fellow Bangalore Entrepreneurs and OpenSource fans.\n\nBy containerizing the storage functionality, the OpenEBS delivers the core storage capabilities like block-layout and its management, data protection, consistency, and availability as a “microservice”, thus bringing the advantages of containers to the storage volumes. Containerization also helps in dynamic provisioning at scale, scale-up/down the storage cluster, monitoring, ease of upgrades, etc.\n\nWe understand that generic orchestration engines will not solve all the problems of storage orchestration. We are augmenting container orchestration engines with the storage intelligence by the OpenEBS orchestration layer — maya. In line with this vision, the 0.2 release has implemented m-apiserver that acts as an interface to the Volume Plugin drivers to provision the storage, while it takes care of interacting with the Container Orchestration Engine to find the right placement for the Storage Containers (VSM).\n\nOpenEBS 0.2 can be used to setup an Amazon EBS-like Block Storage Service for your containers, and consume block storage for your Stateful applications.\n\n![OpenEBS 0.2 setup](https://cdn-images-1.medium.com/max/800/1*itiDxdwyTmdd9VsIYwFYiA.png)\n\nThe Ops team can easily setup an OpenEBS cluster using Bare Metal Machines or VMs, just like setting up Kubernetes Cluster. The Kubernetes minion nodes should be configured with the FlexVolume OpenEBS Driver (openebs-iscsi) to use OpenEBS Storage.\n\nThe DevOps or the Developers can configure Storage to their Stateful Apps from OpenEBS cluster by having their Application intent files point to OpenEBS Maya API Server.\n\nOpenEBS is a completely OpenSource Project that is being driven by the feedback received from the community. In our next milestone 0.3, we are working towards making OpenEBS hyper converged with Kubernetes. Enhance the capabilities around Core Storage for using an additional type of disks for storing block data, providing the controls to the user to perform backup/restore to Amazon S3, etc., Do take a moment to check out our [Project Tracker](https://github.com/openebs/openebs/wiki/Project-Tracker).\n\nIf you are as cautious and skeptical about technology as me, then seeing is believing. To help you quickly get started, we have created Kubernetes and OpenEBS 0.2 Vagrant Boxes.\n\n[Try OpenEBS 0.2 today](https://github.com/openebs/openebs/blob/master/k8s/dedicated/tutorial-ubuntu1604-vagrant.md) and let us know what you think!\n\nHangout with us and help us with your valuable feedback at [slack.openebs.io](http://slack.openebs.io)\n","notHasFeatureImage":false,"slug":"openebs-sprinting-ahead,-0.2-released"},{"id":136,"title":"Software Defined Storage — finally","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"11-03-2017","tags":["DevOps","Docker"],"excerpt":"In this blog, I’ll discuss what went wrong — why we didn’t achieve the promise of software-defined storage — and why software-defined storage is now, finally, possible.","content":"\nMany years ago there was a flowering of what we called software-defined infrastructure. Those of us at the forefront of the trend were no doubt encouraged by the success of Martin Casado and the Nicera team who quickly went from Ph.D. thesis to $1.26bn purchase by VMware thanks to the promise of software-defined *networking*. In hindsight we were misguided, but a handful of us built companies built on the premise — why not *storage*?\n\nToday we finally have the ingredients in and around the storage industry to achieve what we all were shooting for 5–7 years ago. I’m so encouraged by these enabling trends that I’ve gotten back into storage even as it has fallen out of investor favor. Cutting-edge users are now achieving the cost savings and boost to their agility we all promised — and truly believed — were around the corner years ago.\n\nIn this blog I’ll discuss what went wrong — why we didn’t achieve the promise of software-defined storage — and why software-defined storage is now, finally, possible.\n\nThere are at least a handful of reasons that storage failed to achieve the true promise of software-defined storage — that we did not live up to the radical flexibility of virtualized compute for example, where compute jobs were largely freed from their underlying hardware:\n\n1. The lack of portability of the controllers themselves\n2. The inherent stickiness of the underlying data\n3. Inability to deliver performance in a dynamic environment\n4. Immature common standards\n\n****Controller portability and data stickiness:****\n\nStorage controllers have to be able to deliver the policies you want — share these folders with these thousands of people for example — while keeping the underlying data safe irrespective of hardware failure, bit rot, user error, and more.\n\nFor a couple of reasons we in the industry at the time were unable to make our controllers truly software-like in the way that they were deployed in data centers: one, we didn’t sufficiently separate the policy engine or storage scheduler from the care and feeding of the underlying data on disk; two, in part because of this as well as the way that hardware drivers interact with operating systems, our storage controllers tended to need to be bare metal installs, and did not run well in a virtual machine. Perhaps more importantly, because our controllers needed to run on the same systems where the data was stored, they could only ever be as fluid and dynamic as the data itself. In other words, we were chained to our hardware in part by the physics of reading, transporting, and writing data.\n\n****Performance amidst dynamism:****\n\nInformation technology design is often a game of pass the bottleneck. For any given system there is one bottleneck or constraint that limits much of overall performance and hence that throttles the ability of the application to serve user needs. Historically the bottleneck has often been storage — and the rise of virtualization just tightened further this bottleneck thanks to the I/O scrambling that hypervisors perform on the read/ write patterns of the applications running on them.\n\nAnd yet if you are to treat storage — and storage controllers — as software that itself can be dynamically provisioned and moved about then you must be able to deliver performance from underlying systems irrespective of where the controller software is located. Perhaps more importantly, you need to be able to interpret the requirements of the applications themselves and deliver the combination of IOPS and latency they need to meet the requirements of their end-users.\n\nThis problem has aspects of the traveling salesman problem, which is to say, it is not entirely solvable. The way it has been solved in practice is that storage has remained bound to particular sets of typically hugely over provisioned hardware and the combination of applications and underlying storage has itself had to be controlled through affinity rules in the compute schedulers from VMware and OpenStack and others. In the container world, for the most part, we have dealt with the challenges of delivering storage performance to so-called stateful containers **by not having stateful containers**. The vast majority of containers that are deployed don’t actually rely upon data storage in the way that databases for example do. Those containers that are stateful typically tightly couple the underlying storage to containers, thereby removing much of the dynamism and ease of management that was a primary point in moving to containers in the first place.\n\n****Immature common standards****\n\nAs if the inherent difficulty of somehow addressing QoS for storage, while allowing controllers themselves to become more flexible, was not hard enough, the industry structure of IT 4–5 years ago itself made it more difficult to deliver software-defined storage. VMware tried hard to get everyone on the same page via their VASA APIs, which was a way to pass information about applications to the storage and for the storage to essentially sign-up for the performance needed, however, this effort ended up making less transparent and more proprietary the DNA, or operating instructions, of the software-defined data center. Perhaps because these operating instructions themselves were so opaque they never caught on in software-defined data centers.\n\nToday the Kubernetes community is probably our best shot for having a set of commonly accepted application definitions that flow into the infrastructure, to actually deliver software-defined infrastructure. In this case, the DNA is human readable YAML and is managed by a set of open source technologies.\n\nQuick note — pod and resource definitions are not fully fleshed out by Kubernetes for storage. So you can do some basic things, such as limiting the amount of storage by user or application or pod, however storage specific QoS is not yet supported in these definitions. This is a work in progress.\n\nNonetheless, what is possible today gives a good idea of what is coming. For example:\n\n![human readable code example](/images/blog/software-defined-storage-finally-example-code.png)\n\nAs you can see, very simple, human readable, and change controllable easily via GitHub or other systems, which is fundamental to achieving a high degree of automation and control.\n\n**The unevenly distributed future**\n\nWhile this blog and self-assessment of where we got to in software-defined storage may be a bit depressing, there are signs of hope.\n\nWithin the Kubernetes community, for example, hardy pioneers with deep technical expertise such as Pearson are using solutions like StackStorm and much else to build truly developer defined infrastructures that include the use of containers for stateful workloads.\n\nWhat I learned from Pearson and other StackStorm users led me to look for storage intellectual property — and the teams that built it — that could enable the storage freedom promised by software-defined Storage. Specifically, I went looking for technologies that could virtualize or containerize storage controllers while somehow ensuring the delivery of QoS.\n\nIn CloudByte I found a solution that today delivers fine grained control of QoS via an architecture that features virtualized storage controllers. What this means is that the controllers themselves can be live migrated, for example, while continuing to serve storage. This is how CloudByte delivers non-disruptive upgrades — and it also means that you can migrate pieces of your data center from on-premise to the cloud and back — with confidence. However, without QoS controls moving your controllers around would be madness — and so the deep understanding of QoS and the ability to set QoS SLAs by the user or by volume for example is crucially important.\n\n![Storage controller migrating from one site to another](/images/blog/storage-controller-migrating-from-one-site-to-another.png)\n\nIn the above image, I show a storage controller migrating from one site to another.\n\nAnd what CloudByte does for today’s primarily scale-up workloads — largely “pets” — our emerging open source project called OpenEBS will do for emerging scale-out workloads — so called cattle — through “containerized storage for containers.”\n\nIn short — as we plan to further illustrate through use case stories especially of hybrid cloud and backup use cases — in CloudByte / OpenEBS I’ve found not just the building blocks but the first instantiations of the future we all dreamed of several years ago.\n\nTime to reboot — and re-energize; storage now has the ingredients needed to unshackle users and enable them to achieve much more dynamic IT while keeping control of their data. Join me by taking a look at CloudByte and OpenEBS today. I look forward to your feedback.\n","notHasFeatureImage":true,"slug":"software-defined-storage-—-finally"},{"id":137,"title":"Using OpenEBS to build a true on-premise container service","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"28-02-2017","tags":["AWS EBS","Container as a Service","OpenEBS","Persistent Storage","Kubernetes"],"excerpt":"The top questions that could be lingering on the enterprise architect mind in an enterprise are","content":"\nThe top questions that could be lingering on the enterprise architect mind in an enterprise are\n\n- How do I build a true container service in my enterprise?\n- Do I really have to depend on the popular clouds such as Google, AWS, DigitalOcean to start my container journey?\n\nBuilding container services on AWS or GCP is practical today. These cloud platforms provide the infrastructure to start building your container platform. You get VMs for hosting K8s minions and masters and persistent disks (EBS disks or GP disks) to persist the block storage of your applications. If you want to build a similar container service in your enterprise on-premise data center, what are the choices for container infrastructure?\n\nWell, the troubling infrastructure piece to build the true container service in your enterprise is the persistent storage for your applications. No surprise there. VMs or bare metal can be deployed using well-known tools such as OpenStack and KVM, but for deploying EBS or GPD equivalent, you would need something like OpenEBS.\n\nOpenEBS helps the enterprises to build an AWS EBS equivalent or GPD equivalent platforms on-premise. OpenEBS is architected to ease the provisioning and management of persistent volumes at scale. It will have most of the elements of the popular AWS EBS. I had discussed the comparison between AWS EBS and OpenEBS with few community members in our last meetup. The slides are posted at [https://www.slideshare.net/OpenEBS/openebs-containerized-storage-for-containers-meetup-2](https://www.slideshare.net/OpenEBS/openebs-containerized-storage-for-containers-meetup-2)\n\nContainerization of block storage volumes gives the benefit of flexible storage upgrade schedules, treating the storage volume as part of your K8s POD, etc. However, there is more to consider as benefits of OpenEBS. OpenEBS is “Open Source” EBS. A quick feature comparison is as follows\n\n![Comparing AWS EBS & OpenEBS](https://cdn-images-1.medium.com/max/800/1*uu_mIhdqobjf3ftNOtf8KQ.png)\n\nNext, provisioning and consuming persistent volumes through OpenEBS is very similar and simple to that of AWS EBS\n\nIn AWS EBS a user creates and attaches a disk to an EC2 instance. The nuances of underlying storage protocols are hidden underneath and not exposed to the consumer/user.\n\n![Connectivity in AWS EBS](https://cdn-images-1.medium.com/max/800/1*zShnxODcXjTNu-X-qsJa5g.png)\n\nFlow of provisioning and consuming persistent block volumes on AWS EBS\nIn OpenEBS, it is very similar. Once the OpenEBS volumes are expressed as intent in the application YAML config file, the volumes are automatically created on the OpenEBS platform, mounted on K8s minions, and the persistent storage is made available to the application.\n\nExample of OpenEBS volumes getting consumed through iSCSI:\n\n![Connectivity in OpenEBS](https://cdn-images-1.medium.com/max/800/1*Mh9MzX5a_YbV9K_LR8EynA.png)\n\nAn example of OpenEBS volumes getting dynamically provisioned and consumed using k8s-openebs provider is shown below\n\n[https://github.com/openebs/openebs/blob/master/k8s-demo/my-nginx-pod-on-openebs.yaml](https://github.com/openebs/openebs/blob/master/k8s-demo/my-nginx-pod-on-openebs.yaml)\n\nThe next comparison of OpenEBS to AWS EBS is how the volume snapshots are managed for data protection.\n\n![Snapshots on AWS EBS vs OpenEBS](https://cdn-images-1.medium.com/max/800/1*elAnAeYarCwxeCEyXv_Xow.png)\n(***Comparing snapshots management on OpenEBS to that of AWS EBS***) \n\nAs shown above, using OpenEBS, you will have a standard S3 snapshot upload capability so that you can choose your S3 provider. With on-premise S3 technologies like Minio, you can have the container backup infrastructure also as on-prem.\n\nOn the advanced features comparison, OpenEBS steps up to match AWS EBS. Granular QoS control, advanced IOPs management features such as using burst and credit IOPS are some of the features that will be home at OpenEBS\n\n![Other features comparison of AWS EBS & OpenEBS](https://cdn-images-1.medium.com/max/800/1*1WGi8-GdTamykwgnJ0lqjw.png)\n\nBuilding reliable storage platforms in open source is a hard thing. With all the expertise in serving large enterprises, the team seems to be up for the challenge. We hope to see the increasing levels of community engagement in the months to come!\n","notHasFeatureImage":true,"slug":"using-openebs-to-build-a-true-on-premise-container-service"},{"id":138,"title":"Torus (from CoreOS) steps aside as Cloud Native Storage Platform. What now?","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"20-02-2017","tags":["DevOps","Docker","Storage Containers","Kubernetes"],"excerpt":"Torus (from CoreOS), that aimed at providing container-native distributed storage announced that it would discontinue development due to lack of traction.","content":"\n[Torus (from CoreOS)](https://github.com/coreos/torus), which aimed at providing container-native distributed storage announced that it would discontinue development due to lack of traction. It just goes to show the dynamics of the open source projects and how business needs could influence them.\n\nTorus attempted to solve one of the core infrastructure problems, and like any projects that aim to solve the infrastructure problems — compute, network or storage, are slightly harder problems to crack with only a handful of people really venturing into solving them. Maybe, this inertia is also due to the fact that, when infrastructure fails, hell breaks loose — we are all too familiar with those network outages and data loss situations where Database teams spend days recovering data.\n\nHarder problems require renewed focus, at least in the initial stages to get the product off the ground for the basic use case. The entry barrier must be minimal, if not totally absent. Once the core value is established, in terms of ease of use or scale and just solving a problem in an intuitive way, it graduates into the adoption phase.\n\nWith containers changing the landscape of applications, they pose an additional set of requirements on the storage platform, which are covered in my earlier post — [Emerging Storage Trends for Containers](/blog/emerging-storage-trends-for-containers). *Any container-native storage platform developed today should be distributed and natively integrated into container orchestration platforms like Kubernetes*. Torus got these aspects right.\n\nWhile suspending further Torus development, I find it interesting that the committers at CoreOS mentioned Rook.\n\n[Rook](https://github.com/rook/rook), like [GlusterFS-container](https://github.com/gluster/gluster-containers), are not container-native, they provide wrapper projects that make it easy to deploy alongside Kubernetes. In the short-term towards the journey of taking containers into production, this retro-fitting approach may be OK! If there is enough traction to refactor the existing projects (Ceph) as we move forward.\n\nBut as we saw with compute virtualization paving the way for software-defined storage, there is a need for container defined storage. PortWorx and OpenEBS alone currently come close to being container-native storage platforms, built ground-up using containers by teams that have built enterprise and cloud storage platforms.\n\nOpenEBS is a distributed block storage that supports both hyper-converged and dedicated deployment models. However, the implementation of the block storage with OpenEBS follows a different approach for block replication (forked out of [Rancher Longhorn](https://github.com/rancher/longhorn)), which optimizes the meta-data processing of the distributed blocks.\n\nOpenEBS also inherits the data protection capabilities, pushing backups onto Amazon S3. We are currently working on EBS like API for provisioning the storage from K8s. With its cloud integration capabilities, we envision the capabilities supported by OpenEBS to enable our customers to build On-prem container environments that are compatible to be moved into the Cloud and vice-versa.\n\nOpenEBS Storage is delivered through VSMs which are essentially containers — frontend and backend, that can be scheduled by Kubernetes along with the Application Pods. The storage specific parameters are fed into the scheduling algorithms through our OpenEBS storage orchestration layer — maya. This container mode of deployment also reduces the burden of managing a separate storage network. The building blocks of the OpenEBS were covered in this post — [OpenEBS building blocks](/blog/openebs-building-blocks-rancher-longhorn)\n\nIf you were planning to contribute to Torus or just interested in getting your hands dirty with container-native storage platform, please take a closer look at [OpenEBS](https://github.com/openebs/openebs), which is completely open source.\n","notHasFeatureImage":true,"slug":"torus-(from-coreos)-steps-aside-as-cloud-native-storage-platform.-what-now?"},{"id":139,"title":"Deployment modes of OpenEBS","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"13-02-2017","tags":["Container","Docker","Kubernetes","OpenEBS","Storage For Containers"],"excerpt":"OpenEBS supports two modes — Hyper-converged and dedicated. The deployment mode really depends on where and how you want to use OpenEBS.","content":"\nOpenEBS supports two modes — Hyper-converged and dedicated. The deployment mode really depends on where and how you want to use OpenEBS. If you are adding block storage capability to existing Kubernetes minions, hyper-converged mode is most desired, so that you can use the existing hardware as is. If the desire is to get a full fledged EBS type functionality to your on-premise cloud or container needs, then dedicated storage servers for OpenEBS is a better choice.\n\nIn the hyper-converged mode, OpenEBS Maya hooks into the K8s master and minions, hooking into scheduling algorithms for creating OpenEBS VSMs. When used in dedicated mode, the provisioning API are exposed via the OpenEBS Maya master. Dynamic Provisioning of the storage can be enabled using the volume plugin drivers or use the EBS volume plugin as in the case with K8s.\n\n![OpenEBS — Hyper-converged mode](https://cdn-images-1.medium.com/max/800/1*MxM5MmWCB_5mmy7A5bor6Q.png)_(OpenEBS — Hyper-converged mode)_\n\n![OpenEBS — Dedicated mode](https://cdn-images-1.medium.com/max/800/1*MAbRf5rJfv8w_OvZz02q7g.png)_(OpenEBS — Dedicated mode)_\n\nIn both modes, Flannel plays an important role in OpenEBS deployment for storage networking. A lot of issues are yet to be discovered in this area.. yes, this is just a start.\n","notHasFeatureImage":true,"slug":"deployment-modes-of-openebs"},{"id":140,"title":"Containerization meetup - Containers for storage too","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"15-01-2017","tags":["Containerized Storage","Docker","Kubernetes","OpenEBS","Storage Containerization"],"excerpt":"I had the opportunity to talk to a very good group of technologists, DevOPs users in the Digital Ocean Containerization Meetup . Docker 1.13 details and Kubernetes deployment tips were fantastics.","content":"\nI had the opportunity to talk to a very good group of technologists, DevOPs users in the [Digital Ocean Containerization Meetup](https://www.meetup.com/DigitalOceanBangalore/events/236353004/) . Docker 1.13 details and Kubernetes deployment tips were fantastics.\n\nI talked about the containerization for storage. The presentation slides are published [here](http://www.slideshare.net/UmasankarMukkara/openebs-containerized-storage-for-containers). Slide #6 talks about the advantages when the storage volumes are containerized, just like the advantages when the applications are containerized. In the storage containerization, the storage software becomes a micro service. Storage protocol (iSCSI in this case), replication, QoS, Encryption are micro services that are better managed and served when containerized. This form of containerization leads to a true non-disruptive storage upgrade possibility in production.\n\nImagine, there are thousands of storage volumes in production serving thousands of containerized applications. If a storage upgrade has to happen that actually is needed only few volumes, such as a special snapshot or encryption feature or bug fix, then only those containers need to be upgraded. The maintenance window scheduling will be the simplest in such scenarios. To make this possible, OpenEBS abstracts the storage functionality into the user space (slide #7) and each volume is dedicated with a separate storage process in the form of containers… Micro services architecture to the storage, delivered.\n\nI also talked about the building blocks of OpenEBS.\n\nOpenEBS uses or builds on:\n\n- Docker — for achieving containerization. [Jiva](https://hub.docker.com/r/openebs/jiva/) is the docker image\n- Longhorn Rancher — For basic storage replication and distributed scale-out needs. QoS, encryption comes from here.\n- Nomad — Clustering capability among storage hosts as well as storage pods or VSMs is served by Nomad\n- Consul — for the cluster db\n- Flannel — for the container networking needs. We use the flannel’s intelligence for picking up the IP addresses for VSMs automatically, VLANS and other networking stuff. The database portion of flannel is not needed as OpenEBS has the centralized config db for Maya (consul implementation)\n\n![OpenEBS Building Blocks and Integration](/images/blog/containerization-meetup-building-blocks.png)\nAs far as the integration points of OpenEBS are concerned, the provisioning in integrated into k8s. The provisioning of the OpenEBS can be done through the [K8s iSCSI volume interface](https://kubernetes.io/docs/user-guide/volumes/#iscsi) or through the [K8s AWS EBS interface](https://kubernetes.io/docs/user-guide/volumes/#awselasticblockstore). Yes, OpenEBS exposes AWS EBS API. Any orchestration layer or application that knows to connect to and use AWS EBS, will work with OpenEBS.\n\nAs we progress from this early stage, we look forward to work closely with the Docker and k8s user communities.\n","notHasFeatureImage":true,"slug":"containerization-meetup---containers-for-storage-too"},{"id":141,"title":"Emerging Storage Trends for Containers","author":"Kiran Mova","author_info":"Contributor and Maintainer OpenEBS projects. Chief Architect MayaData. Kiran leads overall architecture & is responsible for architecting, solution design & customer adoption of OpenEBS.","date":"11-01-2017","tags":["DevOps","Docker","OpenEBS","Startup","Storage"],"excerpt":"The smiling docker whale is everywhere these days. You either have ridden on it or you want to know how to ride on it. As far as operations teams are concerned the docker whale is just teasing them.","content":"\nThe smiling docker whale is everywhere these days. You either have ridden on it or you want to know how to ride on it. As far as operations teams are concerned the docker whale is just teasing them. In the recent meetups and conferences I attended here, very few hands rise when asked if they have deployed containers in production.\n\nWell, it has taken over a decade and half at least for the shift from servers to virtual machines. And even now, I see a lot of caution in the enterprises to move completely into the public cloud. Though the technology around virtualization is well established, the real return$$ of moving onto public cloud aren’t as great as initially expected.\n\nA similar transitional trend, but at a much faster pace is being seen around the containers. Docker has done a phenomenal job in breaking the entry barrier for playing with containers, by addressing the Container Eco-System components for Developers to quickly build, test and package the applications via containers.\n\n![Interest over time](https://cdn-images-1.medium.com/max/800/1*c8cxwXMmU93xXAiK4Rs0BA.png)\n\nWhile it is very easy to setup and run containers, are dockers really ready for the production work load? The following diagram pretty much sums it up.\n\n(Credit: [http://bicarait.com/2016/11/01/running-your-docker-using-vsphere-integrated-container/](http://bicarait.com/2016/11/01/running-your-docker-using-vsphere-integrated-container/))\n\n![Containers in development & production](https://cdn-images-1.medium.com/max/800/0*nDL6ATRys2vPH8m9.png)\n\nLearning from the recent past, the cloud was led by the compute virtualization, but it wasn’t a true cloud until the network and storage also were fully virtualized, giving way to software-defined Storage and software-defined Networking. These days it is about SDDC, which involves significant improvement in the orchestration of infrastructure and the services running on them. For those of us that have contributed and implemented the cloud, and operated them in production — the journey was exciting and in some instances nail biting!!\n\nI keep hearing developers say, I don’t care about the infrastructure. That’s great, but it just means that someone else is taking care of it, usually the operations team! I wonder if the DevOps is bringing developers closer to operations or Operations closer to Developers? I bet it is the Operations Team getting closer to Developers and making the life of Developers much easier.\n\nThe Developers are excited about the stateless containers or server-less architectures or lambda services. All of these are great architectural patterns, but all these eventually depend in some form on some stateful containers or servers that are running over robust infrastructure with extremely low latency.\n\nThe Operations Teams are the one that need to define this robust infrastructure for running the containers. Unlike VMs that were provisioned (or self provisioned on designated infrastructure) for specific use-cases, the container infrastructure must come with a brain of its own, where the containers are provisioned and moved depending on the load and performance parameters.\n\nAn application or service in the container world, will be a set of interconnected containers (stateful or stateless — much like a K8s Pod), that will require compute, network and storage. And for these Application Pods to be portable, the infrastructure also needs to move along with them. It is time for programmable infrastructure to go mainstream. Just as virtualization paved way for software-defined networks and software-defined storage, we are now seeing the need for container defined network and container defined storage.\n\nFor a recent [Bangalore Docker meetup](http://neependra.net/?p=2141), I took a shot at looking through the various storage startups that are trying to build storage for the containers. The slides are available here : [Emerging Storage Trends for Containers](http://www.slideshare.net/kiranmova/emerging-storagetrendsforcontainers)\n\nThe storage trends can be summarized as follows:\n\n(1) (Slide#22) **Elastic Storage Infrastructure** — The storage can be horizontally scaled, much like the docker hosts in docker swarm. The technology to implement the Elastic Storage is already in production with many vendors supplying **_Software-Defined Scale-out/Distributed Storage_** in the cloud environments.\n\n(2) (Slide #23) **Ease of Accessing the Storage**— The volume of volumes required by the containerized applications will be manyfold compared to volumes used in the VM storage. And the volumes will be more portable. Operations team should be shielded from having to mount/unmount volumes or datastores. **_The integration into the orchestration layers for mounting the storage and providing access to the volumes, without requiring additional software changes will be paramount._** The TCMU/iSCSI, NBD/NFS interfaces are two different approaches with each coming with its own nuances w.r.t isolation vs ease.\n\n(3) (Slide #24) **Hyper Converged Storage** — For storage to be hardened, the developer setups need to be and **_avoid managing silos of infrastructure for compute, network, and storage._** The concept of lightweight storage software will emerge which will mount distributed external storage or cloud storage onto the local machines (with caching).\n\n(4) (Slide #25) **Hybrid Clouds / Storage** — Put the storage in the right place and move it around depending on the demands of the application and economics. A first in this place would be to move the snapshots into S3 when the persistent disks are used for stateful containers. The industry has established that to save money, a mix of clouds is going to be used with the apps moving into different container environments. The storage platform should be able to run alongside different clouds or have the ability to inter-operate. **_Seamless Storage Migration within and across clouds is a must._**\n\n(5) (Slide #26) **Containerized Storage** — One of the best innovations is the way the containers are defined and deployed. The storage software also will reap these benefits by containerizing the storage. **_Version of the storage being used will become another parameter to be defined in the intent specs along with capacity and performance (QoS)._** Containerization also helps with isolation and ease of upgrades in shared environments.\n\n(6) (Slide #27) **Keep up with the core storage innovations** — Flash is already mainstream, with CIOs questioning if they still really need the SAS Tiers. With NVMe, cache is becoming more accessible both in terms of $$ and the performance boost it can provide to the Storage Software. **_The low latency demands from the containerized applications will be guaranteed via the use of NVMe Flash for read/write caching of data that is possibly stored on disks or remote cloud storage._**\n\nIt is euphoric to see the new developments being made in the opensource for the storage, networking, and orchestration layers, apart from the container runtime itself!\n\n2017 may well be year where we will start seeing containers in production as the infrastructure pieces for Containers mature.\n","notHasFeatureImage":true,"slug":"emerging-storage-trends-for-containers"},{"id":142,"title":"OpenEBS Building blocks — Rancher Longhorn","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"06-01-2017","tags":["Container","DevOps","Longhorn","OpenEBS","Docker"],"excerpt":"In the previous blog post, I shared the initial details of OpenEBS. In this post, I will continue to discuss the OpenEBS technology building blocks and touch upon the details of Rancher longhorn and why we chose longhorn.","content":"\nIn the [previous](/blog/openebs-the-containerized-storage) blog post, I shared the initial details of OpenEBS. In this post, I will continue to discuss the OpenEBS technology building blocks and touch upon the details of Rancher longhorn and why we chose [longhorn](https://github.com/rancher/longhorn).\n\nOpenEBS platform contains three core building blocks:\n\n- An orchestration platform, Maya, that works with Kubernetes and manages thousands of volumes with ease.\n- Containerized storage volumes called Virtual Storage Machines or VSMs and\n- Maya managed backing stores or data stores residing either locally on OpenEBS hosts or remotely over network\n\n**_Just to recap, why storage containerization?_** With storage containerization, the storage upgrades are flexible, easy, and effective. The containerization of storage means that the core functionality of storage (like front end protocol ISCSI, snapshotting, replication, backup) is abstracted into a Docker container and managed outside the kernel. A software patch to correct or enhance the replication behavior of a volume does not affect the other volume in the same host. Each of these Docker containers do a specific job of either running [gotgt ](https://github.com/gostor/gotgt)iSCSI or running a [longhorn ](https://github.com/rancher/longhorn)replica. The storage software is built as [Docker image](https://hub.docker.com/r/openebs/jiva/) and is the core/essence of OpenEBS technology. Hence, we named it “Jiva” (meaning “life”, [Wikipedia](https://en.wikipedia.org/wiki/Jiva)).\n\n**_About VSM:_** Virtual Storage Machine is the logical set of storage pods that encapsulates the entire functionality of the life cycle of a volume. The components of an OpenEBS VSM is shown in the below picture.\n\n![Fig: OpenEBS VSM components](https://cdn-images-1.medium.com/max/800/1*-Bl0JyjyNdVe_bp6YI-n6w.png)\n\nA VSM contains as many storage pods as the number of data copies of the volume. Each storage pod has at least one container for replica and optionally has a container for exposing the storage access protocol (iSCSI, NBD etc). We are using a fork of rancher/longhorn software to manage the replication among the storage pods and a fork of gostor/gotgt software to provide iSCSI interface.\n\n## What is longhorn ?\n\nLonghorn is a simplified block storage software, implemented in golang, that stores the entire volume as a single linux sparse file. The sparse files provide thin provisioning behavior. Formatting with QCow2 adds the CoW feature to the data. It is lean and provides an AWS EBS style snapshot functionality. Longhorn has two subcomponents, longhorn controller (LHC) and longhorn replica (LHR). LHC takes care of storage connectivity, replication, rebuild, encryption, etc while LHR does snapshotting, backup, QoS, etc.\n\nLHC and LHR can be deployed in two modes.\n\n- _Hyper-converged container model,_ where LHC and LHR are on the same host as that of compute or Docker Host. TCMU is used for block storage volume drive emulation on Docker Host.\n- _Remote storage model,_ where LHC and LHR are on separate storage host. The compute Docker Host connects to LHC using iSCSI. iSCSI client is used for block storage volume drive emulation on Docker Host.\n\nThese two models are shown below\n\n#### LHC and LHR are on the Docker Host\n\n![Fig. Longhorn deployment mode : Hyper-converged](https://cdn-images-1.medium.com/max/800/1*nlswAfJqgqaWRJpKYLr_jA.png)\n\n_Note: Minimum linux kernel version required for hyper-converged mode is 4.4_\n\n## LHC and LHR are on the remote Storage\n\n![Fig: Longhorn deployment mode : Network storage](https://cdn-images-1.medium.com/max/800/1*wB_PG-Y_jZm8lMmSzKJAww.png)\n\nA third mode is also possible, where LHC runs on the Docker Host and LHR runs on the remote storage host, the discussion about this is for a later day.\n\nLonghorn replica uses 4K as the underlying block size and is a chain of differencing disks among the live data and snapshot data. The backup is done in a AWS-EBS style, where only the changed blocks are copied to the remote location (like S3) using 2M block size.\n\n## Why we chose longhorn for OpenEBS?\n\nWe wanted to implement a simple block storage engine, in user space, that can be containerized. Rancher had spent quite an amount of effort in just doing that. It is written in golang too. We found it to be thin, working, and fit. We integrated gotgt and longhorn for the basic use case of OpenEBS. We are thrilled to find great support from longhorn team in this journey. Thank you, Rancher. Though [openebs longhorn](https://github.com/openebs/longhorn) is forked at the moment from Rancher longhorn, we intend to push back the changes to the mainstream longhorn and contribute there. We plan to add the functionality of flash caching, S3 integration, RDMA support, cache tier-ing to remote storage, etc to longhorn in the days and months to come.\n\nThe next blog post will discuss the deployment modes of OpenEBS with containerized longhorn or jiva.\n","notHasFeatureImage":true,"slug":"openebs-building-blocks-—-rancher-longhorn"},{"id":143,"title":"I’ve become a Chairman — of CloudByte / OpenEBS","author":"Evan Powell","author_info":"Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!","date":"04-01-2017","tags":["DevOps","Docker"],"excerpt":"I’m excited to announce that I recently joined CloudByte, a developer of storage solutions for enterprises and service providers and the originator of the new OpenEBS project.","content":"\nI’m excited to announce that I recently joined CloudByte, a developer of storage solutions for enterprises and service providers and the originator of the new OpenEBS project. While my role is Chairman of the Board of Directors, I’ll be rolling up my sleeves and doing whatever I can to help.\n\nYou might ask — why? And so what?\n\n## 1. **YAML = truth**\n\nWhen I helped popularize the notion of software **defined** open storage years ago as founding CEO of Nexenta — one of the concerns was “who sets the definition?” At the time VMware was pushing themselves for that role. It was a bottoms up — infrastructure up — approach that never fully caught hold.\n\nIt was a pre DevOps approach of doing things that often ended in the old familiar:\n\nThese days the intent of the developer and the requirements of the application are being expressed via flavors of YAML that Kubernetes, Docker Swarm, Mesos, StackStorm, Rancher and others leverage.\n\nCloudByte, via the OpenEBS community that builds on many of the technologies that CloudByte has been building and selling for years, is well positioned to be the preferred solution for containerized storage for containers not just because of the underlying QoS isolation bits — though that heritage is important — but also because we and the OpenEBS community are 100% focused on manifesting the intent and latent requirements of the application.\n\nTake a look at OpenEBS here: [www.OpenEBS.io](http://www.openebs.io/). It is early however already gathering some great contributors and users. Please let us know what you think.\n\n## 2. **I like money**\n\nNo, it isn’t that I’m joining the board for the money. News flash — early stage start-ups don’t have money lying around to pay board members, actually the cash flows the other way :) Rather, CloudByte’s technologies have been proven by demanding enterprises and service providers.\n\nThis existing customer set — and their increasing commitments as confirmed by a far and away record quarter of sales last quarter — gives us a foundation to build upon. We intend to extend our success in the scale up storage space via our flexible business model and superior technology — especially for use cases where our per volume QoS management matters to users.\n\n## 3. **Team time!**\n\nMost importantly, as I’ve come to know the CloudByte team I’ve formed a commitment to them and their vision for the kind of company we can build. I feel honored to support our CEO and co-founder Uma — he’s brilliant, he is stubborn, and he is relentlessly making real a vision for what storage can be — as an **enabler** of the shift towards cloud native and DevOps as opposed to an **inhibitor**. It didn’t hurt that the new board hit it off immediately and that Raj from Fidelity / EightRoads and Sandeep from Nexus have great entrepreneur-friendly reputations as well as massive funds supporting them.\n\nThere will be much more to come as we share our thoughts about strategy and, maybe more importantly, as [OpenEBS](http://www.openebs.io/) community members lay out what they’d like to see in containerized storage.\n\nUntil then — please get in touch. Hang out with us on [the community](http://www.openebs.io/) or just ping me via twitter @epowell101 or otherwise. Let’s work together to make storage — finally — something that does what it is supposed to do despite all the challenges inherent to providing persistence to today’s ever more dynamic environments.\n","notHasFeatureImage":false,"slug":"i’ve-become-a-chairman-—-of-cloudbyte-/-openebs"},{"id":144,"title":"OpenEBS — The containerized storage","author":"Uma Mukkara","author_info":"Contributor at openebs.io, Co-founder & COO@MayaData. Uma led product development in the early days of MayaData (CloudByte).","date":"01-01-2017","tags":["DevOps","Docker","Kubernetes","Rancher","Golang","OpenEBS"],"excerpt":"In the infrastructure space, the compute environment is always the first to lead the change. Docker has brought in the new thinking into every DevOPs administrator and application developer.","content":"\nIn the infrastructure space, the compute environment is always the first to lead the change. Docker has brought in the new thinking into every DevOps administrator and application developer. CIOs across the spectrum are beginning to include Docker into their policies. As the IT environment begins to adopt containerization into mainstream, there are holes that still need to be plugged in, specifically in the storage space.\n\nI am excited to write the first details on OpenEBS ([www.openebs.io](http://www.openebs.io/)) project.\n\nAfter 4 amazing years of building commercial storage products, I recently took a giant leap into the open source model of building infrastructure software. We recently launched OpenEBS project with the goal of building the developer friendly storage using coolest available infrastructure pieces underneath .. be it Docker, k8s, Rancher LongHorn, Nomad, Terraform, etc.\n\n## So, what is OpenEBS, in short ?\n\nOpenEBS offers persistent block storage with the following features:\n\n- Containerized block storage using Docker containers. We call them VSMs or Virtual Storage Machines, a concept similar to k8s PODs.\n- A highly scalable storage orchestration platform that spins the storage volumes seamlessly and manages them effortlessly\n- A simple, yet high performing distributed block storage designed with best caching via the NVMe optimizations\n\n## Why containerized storage?\n\nSimple answer is that even the storage volumes have software associated with them for their regular functions and this software needs to be managed at volume level.\n\n![Monolithic vs Containerized Storage](https://cdn-images-1.medium.com/max/800/1*OoQnpEsGf_ovb5BFnGI8hA.jpeg)\n\nWhen software upgrades happen at storage host level, all volumes' behavior will change simultaneously, which may not be the desired result. Similarly, upgrade maintenance windows may not be the same for all applications or storage volumes. We often observe that it is very difficult to get a convenient window that satisfies all storage volumes or associated applications.\n\nWith containerized storage, storage upgrades becomes simple and easy, just like application upgrades with Docker containers.\n\n## Maya, our new storage orchestration platform\n\nKubernetes, Docker swarm, Rancher cattle, Nomad and other orchestration platforms do a good job of managing the lifecycle of compute containers and initial provisioning of network and storage. However, the storage infrastructure management, when scaled is a big beast in itself. Storage volumes need to be persistent to the application but they need to be volatile in the backend. Storage volumes need to be scheduled on various hosts based on the capacity and IOPS availability and these volumes may need to be moved on the fly as the usage goes up.\n\n_Maya in Sanskrit language means “\\***\\*Magic\\*\\***”. Maya will seamlessly integrate storage management functionality into existing container orchestration layers for provisioning, scheduling, reporting, rolling upgrades etc., and provide storage specific capabilities like data protection capabilities, migrating storage etc.,_\n\n## The building blocks of high performing, distributed block storage:\n\n**_Rancher longhorn:_** We chose to adopt and enhance Rancher longhorn as the basic building block of storage block intelligence in OpenEBS. I will write a separate blog about what is longhorn, it’s features and why we chose longhorn, but in short, longhorn employs a clever and simple approach to container data connectivity, data availability (replication), data protection (snapshot). And longhorn is written in GoLang.\n\n**_Gostor gotgt:_** One of the initial front ends for OpenEBS is of course the iSCSI. We chose gostor/gotgt as a good starting point. OpenEBS plans to add many new capabilities to gotgt like clustering support, performance optimizations etc.\n\n**_Bulk Caching layer through NVMe:_** The caching layer that we see in traditional storage systems is usually small in size. The recent advancements in flash technology made it possible to offer large capacities of flash at affordable prices. Now cost is not a deterrent to have terabytes of low latency flash storage. OpenEBS provides an intelligent caching technology which keeps the hot data in the large NVMe flash layer. Intel’s 3d XPoint is a good fit for this technology.\n\n## Community:\n\nCommunity is paramount. We hope to embrace a lot of friends, advisers, experts in this journey and successfully deliver the OpenEBS promise. Drop by at our [gitter channel](https://gitter.im/openebs/Lobby) and say Hi !\n","notHasFeatureImage":true,"slug":"openebs-—-the-containerized-storage"},{"id":145,"title":"Running through the mud looking skywards","author":"Amit Kumar Das","author_info":"Engineer the DAO","date":"2017-02-11","tags":["Education","Industry","Storage","OpenEBS"],"excerpt":"I always believe, bridging the minds of academia & industry has to go beyond the nuances of theory vs. practical.","content":"\nI knew we would be running through the mud looking skywards for those brightest stars. We were not at all disappointed with our first attempt at _Kuppam Engineering College_, India.\n\nI always believe, bridging the minds of academia & industry has to go beyond the nuances of theory vs. practical. We call ourselves the industry’s core and form the industry’s grey cells. What we need to do is make our problems open to the public. These problems becomes the new practice that needs to be picked up by our _alma mater_. Once selected, it will take no time for these problems to go viral among the students of these institutions. Let these young minds chart their own course; love, enjoy & live every moment when they traverse through this meandering amazon of practical problems. This becomes the fodder for the future theory allowing the current one to rest in peace.\n\nAbove sounds like a grand planning that resembles more like a trickle down effect. Well, the answer is a big ‘No’. Team at [**OpenEBS**](http://openebs.io/) believes there are many faster ways to achieve above in a time bound manner. Hence the formation of various internship programs where [**OpenEBS**](http://openebs.io/) approaches the Universities to get the latter on same plane that the former is passionate about.\n\n**At Kuppam, we could feel the freshness in the air, the disruptions that yearn to come out of the closet. We are all geared to lighten up this brilliance that is currently lying dishevelled.**\n\nOn a closing note, I challenge the myriad number of B.E projects & Ph.D. programs that are purposeless versus the likes of problems that the world wants us to solve. Join me in my effort to solve the problems that matter.\n","notHasFeatureImage":true,"slug":"running-through-the-mud-looking-skywards"}] \ No newline at end of file