Medusa purge task not running

[ClusterJan]

What happened?

When creating a medusa purge task, the task get's started by the k8ssandra-operator, but nothing further happens.

Did you expect to see something different?

The obsolete medusa backups should get deleted.

How to reproduce it (as minimally and precisely as possible):

Have a running K8ssandra cluster with backups that are out of the rotation and run a Purge task:

apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaTask
metadata:
  name: purge-backups-1
spec:
  cassandraDatacenter: kubernetes-1
  operation: purge

`

Environment

• K8ssandra Operator version: 1.10.3
• Cassandra version: 4.1.3
• Medusa version: 0.16.3
• Kubernetes version: 1.24.17

Anything else we need to know?:
Discussed in Discord.

[rzvoncek]

Hi @ClusterJan . I've finally managed to pick this up.
I tried to reproduce this, but I did not manage so far.

• I've created a deployment using the versions from above (aside from the k8s version). I configured maxBackupCount to 3, created 4 backups, created a purge task, and saw the oldest backup disappear.
• On our dogfood cluster we have about 370 backups, so I set the maxBackupCount to 369 and saw the oldest backup disappear.

I'll give it another go, but I'd like to ask for more input.

• What's your maxBackupCount and maxBackupAge set to? You can see this either in the Medusa's config map (the one with key medusa.ini, or in the k8ssandra-operator's Medusa spec.
• You mentioned about 16 backups have missed their deletion, but many backups are present in total?
• I also saw the logs from the operator. I'd be also interesting to see the logs from the Medusa containers of the Cassandra pods. I'm not yet sure what precisely to look for there, so we might just need to to give it an quick look first.

[ClusterJan]

Hey, thank you for taking a more in-depth look into this. Eventually, I was able to take another look as well. As it seems, the purges do work again as intended, but I'm not quite sure what could have caused this issue.

Mainly it was weird, that I wasn't able to see any confirmation of an ongoing purge in the logs, and it seemed that it might have been stuck. But I can confirm that it works in our cluster again and also syncs the medusabackups correctly as well. So I'm going to close this issue. :)

[adejanovski]

Thanks for the update!