Skip to content

Latest commit

 

History

History
276 lines (214 loc) · 10.4 KB

README.md

File metadata and controls

276 lines (214 loc) · 10.4 KB
Raw

fluent-pvc-operator

fluent-pvc-operator is a Kubernetes Operator that aims to dynamically provision PVCs to Pods. This Operator make you possible to attach a disposable PVC to a Pod without issuing PVCs in advance.

Motivation

The issues we want to solve with this Operator are as follows:

  • Provide a way to protect the data in a Pod from sudden death of a Node by persisting the data to a PersistentVolume.
  • Enable to offload filesystem-dependent Pod termination operations to other Pods.
  • Make PVC available for Pods generated from Templates such as Deployment and DaemonSet.

Features

fluent-pvc-operator has the following features:

  • Dynamic PVC Provisioning: Creates a PVC and injects it into the Pod Manifest on Pods creation admission webhook.
  • Sidecar Container Injection: Injects a container definition into the Pod Manifest on Pods creation admission webhook.
  • Unhealthy Pod Auto Deletion: Detects anomalies in the Injected Sidecar Container and automatically deletes the Pod.
  • PVC Auto Finalization: After the Pod is deleted, a Job is automatically issued to process the data in the PVC, and if the Job is successful, the PVC is deleted.

Planned Features

  • Sidecar Container Auto Termination: Terminates the Sidecar Container automatically when the specified Container in the Pod has been terminated. This feature is intended to be used in Job.

Custom Resource Definitions

There are two Custom Resource Definitions that fluent-pvc-operator installs:

Usage

Put fluent-pvc-operator.tech.zozo.com/fluent-pvc-name: <YOUR_DEFINED_FLUENT_PVC> in the labels of your pod, then fluent-pvc-operator processes the pod as a target.

apiVersion: v1
kind: Pod
metadata:
  labels:
    fluent-pvc-operator.tech.zozo.com/fluent-pvc-name: fluent-pvc-sample
  name: your-pod
spec:
  ...

Behaviors

  • On Pod Scheduling
    • Create a PVC for the Pod.
    • Inject the PVC to the Pod Manifest.
    • Inject the Sidecar Container Definition to the Pod Manifest.
  • On Pod Running
    • Monitor the Sidecar Container status.
    • Delete the Pod when the Sidecar Container is terminated with exit code != 0.
  • On Pod Terminated
    • Apply the finalizer Job for the PVC.
    • Delete the PVC when the finalizer Job is succeeded.

Configurations

fluentpvcs.fluent-pvc-operator.tech.zozo.com

name type required? default description
pvcSpecTemplate PersistentVolumeClaimSpec true Template to provision PVCs
pvcFinalizerJobSpecTemplate JobSpec true Template to apply Jobs for finalizing PVCs
pvcVolumeName string true Name of Volume to use PVCs for Pods. Must be a DNS_LABEL and unique within the Pod.
pvcVolumeMountPath string true Path to mount containers as a VolumeMount.Must not contain ':'.
sidecarContainerTemplate Container true Template for Sidecar Container injected into Pods.
commonEnvs []EnvVar false [] Common Environment Variables for all containers
commonVolumes []Volume false [] Common Volumes for all Pods
commonVolumeMounts []VolumeMount false [] Common VolumeMounts for all containers
deletePodIfSidecarContainerTerminationDetected boolean false true Flag to delete Pods when the injected sidecar container termination is detected.

sample

apiVersion: fluent-pvc-operator.tech.zozo.com/v1alpha1
kind: FluentPVC
metadata:
  name: fluent-pvc-sample
spec:
  pvcSpecTemplate:
    accessModes: [ "ReadWriteOnce" ]
    storageClassName: standard
    resources:
      requests:
        storage: 1Gi
  pvcFinalizerJobSpecTemplate:
    template:
      spec:
        restartPolicy: Never
        containers:
          - name: sidecar
            image: alpine:latest
            imagePullPolicy: Always
            command: [echo, finalizer]
            resources:
              limits:
                cpu: '1'
                memory: 1Gi
  pvcVolumeName: fluent-pvc
  pvcVolumeMountPath: /mnt/fluent-pvc
  sidecarContainerTemplate:
    name: sidecar
    image: alpine:latest
    imagePullPolicy: Always
    command: [echo sidecar]
    resources:
      limits:
        cpu: '1'
        memory: 1Gi
  deletePodIfSidecarContainerTerminationDetected: true
  commonEnvs:
    - name: FLUENT_PVC_MOUNT_DIR
      value: /mnt/fluent-pvc
  commonVolumes:
    - name: SOME_SECRET
      secret:
        secretName: some-secret
  commonVolumeMounts:
    - name: SOME_SECRET
      mountPath: /path/to/secret

Installs

$ git clone https://github.com/st-tech/fluent-pvc-operator.git
$ cd fluent-pvc-operator
$ make deploy IMG=ghcr.io/st-tech/fluent-pvc-operator:0.0.1

Requirements

Designs

Programs

  • fluentpvc_controller.go
    • Monitor the Finalizer of all FluentPVCBindings whose Owner Controller is the FluentPVC.
    • Remove the Finalizer from FluentPVC after the Finalizer is removed from all FluentPVCBindings.
  • fluentpvcbinding_controller.go
    • Monitor the Pod, PVC and Job defined in FluentPVCBinding.
    • Update the condition of FluentPVCBinding according to each condition change.
    • Each controller decides what to do according to the condition of FluentPVCBinding.
    • Cannot delete FluentPVCBinding until the PVC Finalizer fluent-pvc-operator.tech.zozo.com/pvc-protection is deleted.
  • pod_controller.go
    • Monitor the Pod defined in FluentPVCBinding.
    • Delete the Pod if the Sidecar Container anomaly is detected.
  • pvc_controller.go
    • Monitor the PVC defined in FluentPVCBinding.
    • Apply the Job to finalize the PVC that the Pod is no longer in use.
    • Delete the PVC when the Job is succeeded.
  • pod_webhook.go
    • Mutate Pods on Pods creation.
    • Creates PVCs and inject the PVC into Pods.
    • Inject the sidecar container definition into Pods.
    • Creates FluentPVCBindings with FluentPVC, Pod, and PVC identities.

Development

Use kind to create local Kubernetes clusters.

Create a Kubernetes Cluster for development

$ kind create cluster
$ make cert-manager

Build fluent-pvc-operator

$ make docker-build

Load the image into the kind cluster

$ make kind-load-image-fluent-pvc-operator

Deploy fluent-pvc-operator

$ make fluent-pvc-operator

Watch the behaviors

$ kubectl apply -f config/samples/fluent-pvc-operator_v1alpha1_fluentpvc.yaml
$ kubectl run --image=alpine:latest --labels fluent-pvc-operator.tech.zozo.com/fluent-pvc-name=fluent-pvc-sample sample-pod -- sh -c 'for i in $(seq 1 60); do sleep 1; echo $i; done'

## You can watch the status changes by the following command.
$ watch -n1 "
echo '=======FluentPVC======='
kubectl get fluentpvc
echo '=======FluentPVCBinding======='
kubectl get fluentpvcbinding
echo '=======PVC======='
kubectl get pvc
echo '=======Job======='
kubectl get job
echo '=======Pod======='
kubectl get pod
echo '=============='
"

Run unit tests

$ make test

These tests are runnable without kind clusters.

Run e2e tests

## Run e2e tests with recreating the kind cluster.
$ make e2e/clean-test

## Run e2e tests on the existing kind cluster.
$ make e2e/test

Examples

The examples directory contains several examples that can be used as a reference for using fluent-pvc-operator.

For log-collection

This example assumes the usecase where the Pod logs are collected by fluentd and sent to Cloud Pub/Sub. The Cloud Pub/Sub used in this case is launched as an Emulator in the same cluster, so there is no need to prepare anything.

Build docker images

$ make examples/log-collection/build

Load docker images into Kubernetes cluster created by kind

$ make examples/log-collection/kind-load-image

Deploy the example manifests

$ make examples/log-collection/deploy

You can deploy manifests with recreating the Kubernetes cluster by kind.

$ make examples/log-collection/clean-deploy

CHANGELOG

Please see the list of releases for information on changes between releases.

License

MIT LICENSE