Backup aren't executed if multiple DCs with the same name exist in a namespace

[adejanovski]

What happened?
Using DC name overrides, it is possible to create multiple DCs with the same name (the cassdc objects will have a different name, but the dc labels will be the same).
In this case, the MedusaBackupJob controller will list the Cassandra pods using just the Datacenter Label as selector, which will return pods from both clusters.
If the other DC doesn't have Medusa configured, the controller will fail the MedusaBackupJob reconcile saying the backup sidecar wasn't found. If the other DC does have Medusa, the backup will be started on the wrong cluster as well as the right one.

Did you expect to see something different?
The backup controller should filter pods by datacenter and cluster name to avoid confusion.

• K8ssandra Operator version:

  v1.10.3

• K8ssandra Operator Logs:

2023-12-08T14:35:46.836Z	ERROR	medusa is not deployed	{"controller": "medusabackupjob", "controllerGroup": "medusa.k8ssandra.io", "controllerKind": "MedusaBackupJob", "MedusaBackupJob": {"name":"backup-2023-12-08t14-29-23-227z","namespace":"testing-cluster-48o2037p"}, "namespace": "testing-cluster-48o2037p", "name": "backup-2023-12-08t14-29-23-227z", "reconcileID": "7576b2d8-3b53-4db8-a49a-6813ec50f62a", "medusabackupjob": "testing-cluster-48o2037p/backup-2023-12-08t14-29-23-227z", "CassandraDatacenter": {"namespace": "testing-cluster-48o2037p", "name": "test-dc1"}, "error": "the backup sidecar was not found"}
github.com/k8ssandra/k8ssandra-operator/controllers/medusa.(*MedusaBackupJobReconciler).Reconcile
	/workspace/controllers/medusa/medusabackupjob_controller.go:194
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235