S3_EXPIRE_DAYS and vault 1.16.3

Done:
* doc: describe deletion marker and how to undo delete markers
* feat: bump vault version to 1.16.3
* feat(kubernetes): add S3_EXPIRE_DAYS

Adds the variable S3_EXPIRE_DAYS for Kubernetes CronJobs.

The idea of this feature is to allow the script to prune expired
snapshot files on the S3 compatible remote storage. Files are considered
expired once they exceed the threshold defined by S3_EXPIRE_DAYS.

This feature is useful for S3 compatible storage where there exist no
lifecycle rules to clean up the storage of expired or old files, such
as:
- cloudscale object storage
- Exoscale simple object storage (SOS)

It is recommended to also configure a "Governance" lock on the files, to
ensure no files are deleted by accident before the defined
S3_EXPIRE_DAYS threshold.

The date manipulation should work even in the busybox environments (e.g.
OpenShift). It simply subtracts seconds.

kubernetes/Dockerfile

@@ -1,6 +1,6 @@
 FROM alpine
 
-ARG VAULT_VERSION=1.13.2
+ARG VAULT_VERSION=1.16.3
 
 COPY vault-snapshot.sh /
 

kubernetes/README.md

@@ -15,5 +15,39 @@ After the snapshot is created in a temporary directory, `s3cmd` is used to sync
 * `S3_URI` - S3 URI to use to upload (s3://xxx)
 * `S3_BUCKET` - S3 bucket to point to
 * `S3_HOST` - S3 endpoint
+* `S3_EXPIRE_DAYS` - Delete files older than this threshold (expired)
 * `AWS_ACCESS_KEY_ID` - Access key to use to access S3
 * `AWS_SECRET_ACCESS_KEY` - Secret access key to use to access S3
+
+## Configuration of file retention (pruning)
+
+With AWS S3, use [lifecycle
+rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html)
+to configure retention and automatic cleanup action (prune) for expired files.
+
+For other S3 compatible storage, ensure to set [Governance
+lock](https://community.exoscale.com/documentation/storage/versioning/#set-up-the-lock-configuration-for-a-bucket)
+to avoid any modification before `$S3_EXPIRE_DAYS`:
+
+```
+mc retention set --default GOVERNANCE "${S3_EXPIRE_DAYS}d" my-s3-remote/my-bucket
+```
+
+On removal by the `vault-snapshot.sh` script, [`DEL` deletion marker
+(tombstone)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-managing.html#object-lock-managing-delete-markers)
+is set:
+
+```
+mc ls --versions my-snapshots/vault-snapshots-2f848f
+[2024-09-09 09:07:46 CEST]     0B X/1031980658232456253 v2 DEL vault_2024-09-06-1739.snapshot
+[2024-09-06 19:39:49 CEST]  28KiB Standard 1031052557042383613 v1 PUT vault_2024-09-06-1739.snapshot
+```
+
+Use [`mc
+undo`](https://min.io/docs/minio/linux/reference/minio-mc/mc-undo.html) to undo
+the `DEL` operation:
+```
+mc undo my-snapshots/vault-snapshots-2f848f/vault_2024-09-06-1739.snapshot
+mc ls --versions my-snapshots/vault-snapshots-2f848f
+[2024-09-06 19:39:49 CEST]  28KiB Standard 1031052557042383613 v1 PUT vault_2024-09-06-1739.snapshot
+```

kubernetes/cronjob.yaml

@@ -32,6 +32,9 @@ spec:
               value: bucketname
             - name: S3_URI
               value: s3://bucketname
+              # leave empty to retain snapshot files (default)
+            - name: S3_EXPIRE_DAYS
+              value:
             - name: VAULT_ROLE
               value: vault-snapshot
             - name: VAULT_ADDR

kubernetes/vault-snapshot.sh

@@ -7,8 +7,22 @@ VAULT_TOKEN=$(vault write -field=token  auth/kubernetes/login role="${VAULT_ROLE
 export VAULT_TOKEN
 
 # create snapshot
-
 vault operator raft snapshot save /vault-snapshots/vault_"$(date +%F-%H%M)".snapshot
 
 # upload to s3
 s3cmd put /vault-snapshots/* "${S3_URI}" --host="${S3_HOST}" --host-bucket="${S3_BUCKET}"
+
+# remove expired snapshots
+if [ "${S3_EXPIRE_DAYS}" ]; then
+    s3cmd ls "${S3_URI}" --host="${S3_HOST}" --host-bucket="${S3_BUCKET}" | while read -r line; do
+        createDate=$(echo "$line" | awk '{print $1" "$2}')
+        createDate=$(date -d"$createDate" +%s)
+        olderThan=$(date --date @$(($(date +%s) - 86400*S3_EXPIRE_DAYS)) +%s)
+        if [ "$createDate" -lt "$olderThan" ]; then
+            fileName=$(echo "$line" | awk '{print $4}')
+            if [ "$fileName" != "" ]; then
+                s3cmd del "$fileName" --host="${S3_HOST}" --host-bucket="${S3_BUCKET}"
+            fi
+        fi
+    done;
+fi