Exascaler-csi-file-driver

Releases can be found here - https://github.com/DDNStorage/exa-csi-driver/releases

Feature List

Feature	Feature Status	CSI Driver Version	CSI Spec Version	Kubernetes Version
Static Provisioning	GA	>= 1.0.0	>= 1.0.0	>=1.18
Dynamic Provisioning	GA	>= 1.0.0	>= 1.0.0	>=1.18
RW mode	GA	>= 1.0.0	>= 1.0.0	>=1.18
RO mode	GA	>= 1.0.0	>= 1.0.0	>=1.18
Expand volume	GA	>= 1.0.0	>= 1.1.0	>=1.18
StorageClass Secrets	GA	>= 1.0.0	>=1.0.0	>=1.18
Mount options	GA	>= 1.0.0	>= 1.0.0	>=1.18

Requirements

• Kubernetes cluster must allow privileged pods, this flag must be set for the API server and the kubelet (instructions):

  --allow-privileged=true
  

• Required the API server and the kubelet feature gates (instructions):

  --feature-gates=VolumePVCDataSource=true,ExpandInUsePersistentVolumes=true,ExpandCSIVolumes=true,ExpandPersistentVolumes=true
  

• Mount propagation must be enabled, the Docker daemon for the cluster must allow shared mounts (instructions)

Installation

1. Clone or untar driver (depending on where you get the driver)

   git clone https://github.com/DDNStorage/exa-csi-driver.git /opt/exascaler-csi-file-driver
   or
   rpm -Uvh exa-csi-driver-1.0-1.el7.x86_64.rpm
   
   docker load -i /opt/exascaler-csi-file-driver/bin/exascaler-csi-file-driver.tar

2. Copy /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver-config.yaml to /etc/exascaler-csi-file-driver-v1.0/exascaler-csi-file-driver-config.yaml

   cp /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver-config.yaml /etc/exascaler-csi-file-driver-v1.0/exascaler-csi-file-driver-config.yaml 
   

Edit /etc/exascaler-csi-file-driver-v1.0/exascaler-csi-file-driver-config.yaml file. Driver configuration example:

exaFS: 10.3.196.24@tcp:/csi             # Full path to EXAscaler filesystem
mountPoint: /exaFS                      # Mountpoint where EXAscaler filesystem will be mounted on the host
debug: true                             # more logs

3. Create Kubernetes secret from the file:

   kubectl create secret generic exascaler-csi-file-driver-config --from-file=/etc/exascaler-csi-file-driver-v1.0/exascaler-csi-file-driver-config.yaml

4. Register driver to Kubernetes:

   kubectl apply -f /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver.yaml

Usage

Dynamically provisioned volumes

For dynamic volume provisioning, the administrator needs to set up a StorageClass in the PV yaml (/opt/exascaler-csi-file-driver/examples/exa-dynamic-nginx.yaml for this example) pointing to the driver. For dynamically provisioned volumes, Kubernetes generates volume name automatically (for example pvc-ns-cfc67950-fe3c-11e8-a3ca-005056b857f8-projectId-1001).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: exascaler-csi-file-driver-sc-nginx-dynamic
provisioner: exa.csi.ddn.com
mountOptions:                        # list of options for `mount -o ...` command
#  - noatime                         #
parameters:
  projectId: "100001"      # Required. Points to EXA project id to be used to set volume quota.
  exaMountUid: "1001"      # Uid which will be used to access the volume in pod. Should be synced between EXA server and clients.
  bindMount: "false"       # Determines, whether volume will bind mounted or as a separate lustre mount.
  exaFS: "10.204.86.114@tcp:/testfs"   # Overrides exaFS value from config. Use this to support multiple EXA filesystems.
  mountPoint: /exaFS       # Overrides mountPoint value from config. Use this to support multiple EXA filesystems.
  mountOptions: ro,noflock

Example

Run Nginx pod with dynamically provisioned volume:

kubectl apply -f /opt/exascaler-csi-file-driver/examples/exa-dynamic-nginx.yaml

# to delete this pod:
kubectl delete -f /opt/exascaler-csi-file-driver/examples/exa-dynamic-nginx.yaml

Static (pre-provisioned) volumes

The driver can use already existing Exasaler filesystem, in this case, StorageClass, PersistentVolume and PersistentVolumeClaim should be configured.

StorageClass configuration

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: exascaler-csi-driver-sc-nginx-persistent
provisioner:  exa.csi.ddn.com
mountOptions:                        # list of options for `mount -o ...` command
#  - noatime                         #

PersistentVolume configuration

apiVersion: v1
kind: PersistentVolume
metadata:
  name: exascaler-csi-driver-pv-nginx-persistent
  labels:
    name: exascaler-csi-driver-pv-nginx-persistent
spec:
  storageClassName: exascaler-csi-driver-sc-nginx-persistent
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 1Gi
  csi:
    driver: exa.csi.ddn.com
    volumeHandle: /exaFS/nginx-persistent
    volumeAttributes:         # volumeAttributes are the alternative of storageClass params for static (precreated) volumes.
      exaMountUid: "1001"     # Uid which will be used to access the volume in pod.
      #mountOptions: ro, flock # list of options for `mount` command

CSI Parameters:

Name	Description	Example
driver	[required] installed driver name " exa.csi.ddn.com"	exa.csi.ddn.com
volumeHandle	[required] EXAScaler server IP and path to existing EXAScaler filesystem	/exaFS/nginx-persistent
exaMountUid	Uid which will be used to access the volume from the pod.	1015
projectId	[required] Points to EXA project id to be used to set volume quota.	100001
exaFS	Overrides exaFS value from config. Use this to support multiple EXA filesystems.	10.0.1.1@tcp:/fs01
mountPoint	Overrides Mountpoint where EXAscaler filesystem will be mounted on the host	/sc-mnt
bindMount	Determines, whether volume will bind mounted or as a separate lustre mount.	true
mountOptions	Options that will be passed to mount command (-o <opt1,opt2,opt3>)	ro,flock

PersistentVolumeClaim (pointing to created PersistentVolume)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: exascaler-csi-driver-pvc-nginx-persistent
spec:
  storageClassName: exascaler-csi-driver-cs-nginx-persistent
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      # to create 1-1 relationship for pod - persistent volume use unique labels
      name: exascaler-csi-file-driver-pv-nginx-persistent

Example

Run nginx server using PersistentVolume.

Note: Pre-configured filesystem should exist on the EXAScaler: /exaFS/nginx-persistent.

kubectl apply -f /opt/exascaler-csi-file-driver/examples/nginx-persistent-volume.yaml

# to delete this pod:
kubectl delete -f /opt/exascaler-csi-file-driver/examples/nginx-persistent-volume.yaml

Updating the driver version

To update to a new driver version, you need to follow the following steps:

1. Remove the old driver version

kubectl delete -f /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver.yaml
kubectl delete secrets exascaler-csi-file-driver-config
rpm -evh exa-csi-driver

2. Download the new driver version (git clone or new ISO)
3. Copy and edit config file

   cp /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver-config.yaml /etc/exascaler-csi-file-driver-v1.1/exascaler-csi-file-driver-config.yaml
   

4. Update version in /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver.yaml

          image: exascaler-csi-file-driver:v1.1

5. Load new image

docker load -i /opt/exascaler-csi-file-driver/bin/exascaler-csi-file-driver.tar

6. Apply new driver

kubectl create secret generic exascaler-csi-file-driver-config --from-file=/etc/exascaler-csi-file-driver-v1.1/exascaler-csi-file-driver-config.yaml
kubectl apply -f /opt/exascaler-csi-file-driver/deploy/kubernetes/exascaler-csi-file-driver.yaml

Troubleshooting

Logs can be found in /var/log/containers directory. To collect all driver related logs, you can use the following command

mkdir -p /tmp/exascaler-csi-file-driver-logs/
cp /var/log/containers/exascaler-csi-* /tmp/exascaler-csi-file-driver-logs/