Update Cassandra users when corresponding secrets are updated

[adejanovski]

cass-operator manages the creation of users in Cassandra based on secrets that are referenced in the CassandraDatacenter spec:

  superuserSecretName: test-superuser
  users:
    - secretName: test-reaper
      superuser: true

In order to support credentials rotation, we need to monitor these secrets and update the credentials in Cassandra accordingly through the management api as part of the reconcile.
We also need to support recreating the credentials in case of a remote restore using Medusa. In this case we do not restore the system_auth keyspace, but since we're changing the token assignments the actual entries in the sstables may not end up on the nodes replicating the associated tokens. We need to force the recreation of the users through the same mechanism, by doing a bogus update on the user secrets in order to trigger the operation.

Definition of Done

Preview Give feedback

1. user related secrets are watched by cass-operator and trigger a reconcile in case of generation change
2. when a reconcile is triggered by a secret change, the corresponding user is updated in Cassandra with the new password

[burmanm]

All of this should be implemented already in cass-operator. cass-operator watches secrets set in superUserSecretName and Users.

https://github.com/k8ssandra/cass-operator/blob/master/internal/controllers/cassandra/cassandradatacenter_controller.go#L231

[adejanovski]

Dang! It's already handled indeed, and I only learn this now 😅
One less thing to implement, nice!

[burmanm]

I had planned to make some changes to the user process, through

https://github.com/k8ssandra/cass-operator/pull/358/files

Since k8ssandra-client has users modification features (just lacks TLS support).