Purging backups using MedusaTask is not cleaning backups from Minio which contains SSTables with empty "bti-Rows.db" files

[Arun-Trichy]

What happened?
Created some backups using MedusaBackupSchedule CRD, and created a MedusaTask CRD with "operation: purge" to remove old backups based on max_backup_age & max_backup_count configurations.

Did you expect to see something different?
All old backups should get removed as per max_backup_age & max_backup_count configurations. But could see some backups still present in Minio and all the backups present are having a common file "bti-Rows.db" with size 0.

How to reproduce it (as minimally and precisely as possible):
Create backup using k8ssandra-operator and ensure that there is SSTables with size 0 or empty files. Run the MedusaTask to purge old backups and observe Minio.

Environment
used Minio for storing the backups, followed https://docs.k8ssandra.io/tasks/backup-restore/

• K8ssandra Operator version: 1.15.0
• Kubernetes version information:
  Client Version: v1.29.1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.1
• Kubernetes cluster kind: used kind https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster
• Manifests:

kc get k8ssandracluster demo -n medusa -o yaml
apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  annotations:
    k8ssandra.io/initial-system-replication: '{"dc1":3}'
  creationTimestamp: "2024-05-21T11:37:58Z"
  finalizers:
  - k8ssandracluster.k8ssandra.io/finalizer
  generation: 3
  name: demo
  namespace: medusa
  resourceVersion: "1440488"
  uid: 7d8fa01a-a543-41f6-b2ce-6f8a6d113b94
spec:
  auth: true
  cassandra:
    datacenters:
    - config:
        jvmOptions:
          gc: G1GC
          heapSize: 4096M
      metadata:
        name: dc1
      size: 3
      storageConfig:
        cassandraDataVolumeClaimSpec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
          storageClassName: standard
    serverType: dse
    serverVersion: 6.8.41
    superuserSecretRef:
      name: demo-superuser
  medusa:
    storageProperties:
      backupGracePeriodInDays: 0
      bucketName: medusafull
      concurrentTransfers: 1
      host: minio.minio.svc.cluster.local
      maxBackupAge: 1
      maxBackupCount: 2
      multiPartUploadThreshold: 104857600
      port: 9000
      prefix: test
      secure: false
      storageProvider: s3_compatible
      storageSecretRef:
        name: medusa-bucket-key
      transferMaxBandwidth: 50MB/s
  secretsProvider: internal
status:...


kc get medusabackup
NAME                                          STARTED   FINISHED   NODES   FILES   SIZE        COMPLETED   STATUS
medusa-differ-backup-2m-schedule-1716293280   18h       18h        3       892     1.17 MB     3           SUCCESS
medusa-full-backup-2m-schedule-1716293280     18h       18h        3       836     914.94 KB   3           SUCCESS

kc get medusabackupjob
NAME                                          STARTED   FINISHED
medusa-differ-backup-2m-schedule-1716292080   18h       18h
medusa-differ-backup-2m-schedule-1716292200   18h       18h
medusa-differ-backup-2m-schedule-1716292320   18h       18h
medusa-differ-backup-2m-schedule-1716292440   18h       18h
medusa-differ-backup-2m-schedule-1716292560   18h       18h
medusa-differ-backup-2m-schedule-1716292680   18h       18h
medusa-differ-backup-2m-schedule-1716292800   18h       18h
medusa-differ-backup-2m-schedule-1716292920   18h       18h
medusa-differ-backup-2m-schedule-1716293040   18h       18h
medusa-differ-backup-2m-schedule-1716293160   18h       18h
medusa-differ-backup-2m-schedule-1716293280   18h       18h
medusa-full-backup-2m-schedule-1716292080     18h       18h
medusa-full-backup-2m-schedule-1716292200     18h       18h
medusa-full-backup-2m-schedule-1716292320     18h       18h
medusa-full-backup-2m-schedule-1716292440     18h       18h
medusa-full-backup-2m-schedule-1716292560     18h       18h
medusa-full-backup-2m-schedule-1716292680     18h       18h
medusa-full-backup-2m-schedule-1716292800     18h       18h
medusa-full-backup-2m-schedule-1716292920     18h       18h
medusa-full-backup-2m-schedule-1716293040     18h       18h
medusa-full-backup-2m-schedule-1716293160     18h       18h
medusa-full-backup-2m-schedule-1716293280     18h       18h

kc get medusatask
NAME                   AGE
purge-backups-1        17h
purge-backups-1-sync   17h
purge-backups-3        17h
purge-backups-3-sync   17h

kc get medusatask purge-backups-3 -o yaml
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaTask
metadata:
  creationTimestamp: "2024-05-21T12:20:43Z"
  generation: 1
  name: purge-backups-3
  namespace: medusa
  ownerReferences:
  - apiVersion: cassandra.datastax.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CassandraDatacenter
    name: dc1
    uid: 186f42eb-a4e4-4351-b7cb-15fb42de81fe
  resourceVersion: "1272569"
  uid: 56dbc707-1537-4a68-9432-56dc179d8bd7
spec:
  cassandraDatacenter: dc1
  operation: purge
status:
  finishTime: "2024-05-21T12:21:13Z"
  finished:
  - podName: demo-dc1-default-sts-0
    totalObjectsWithinGcGrace: 693
  - podName: demo-dc1-default-sts-2
    totalObjectsWithinGcGrace: 631
  - podName: demo-dc1-default-sts-1
    totalObjectsWithinGcGrace: 673
  startTime: "2024-05-21T12:20:58Z"

kc get medusatask purge-backups-3-sync -o yaml
apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaTask
metadata:
  creationTimestamp: "2024-05-21T12:21:13Z"
  generation: 1
  name: purge-backups-3-sync
  namespace: medusa
  ownerReferences:
  - apiVersion: cassandra.datastax.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: CassandraDatacenter
    name: dc1
    uid: 186f42eb-a4e4-4351-b7cb-15fb42de81fe
  resourceVersion: "1272615"
  uid: b02ea9d0-ca79-402e-9025-99f806826cee
spec:
  cassandraDatacenter: dc1
  operation: sync
status:
  finishTime: "2024-05-21T12:21:28Z"
  finished:
  - podName: demo-dc1-default-sts-2
  startTime: "2024-05-21T12:21:28Z"

Minio contents of one of the Cassandra nodes:


Medusa sidecar container logs showing Purging operation removing all files except "bti-Rows.db"

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: K8OP-23