When CRDs are not upgraded, ReplicatedSecrets may copy infinitely

[Miles-Garnsey]

What is missing?

We now use the DropLabels field in the replicatedSecret to ensure that a prefixed secret copied into the same namespace as it's original (and the ReplicatedSecret controlling the replication) does not continue to replicate infinitely.

But if a CRD is not properly upgraded so the DropLabels field exists, or if a user accidentally misconfigures a ReplicatedSecret, then infinite replication can still occur. We should put a safeguard in place to ensure that a secret replicating into it's own namespace will never replicate if doing so would cause an infinite loop.

It is worth noting that we cannot assure this in the case that a ReplicationTarget has been defined with a non zero K8sContext which nonetheless points back to the local cluster. In this case, there is no way for us to tell that the K8sContext is the local one, and infinite replication might still occur.

Why do we need it?

We've seen this bug happen once before and it might be dangerous to the cluster given that it continues to create so many copies. It should not happen in ordinary operations, but the extra safety is justified.

Environment

• K8ssandra Operator version:

  Insert image tag or Git SHA here

Anything else we need to know?:

[burmanm]

We can replicate the secret to its own destination, that's no problem. A copied secret should just have an annotation that indicates it's a copy and thus it would never be used as a source.

[adejanovski]

the problem is that these secrets are replicated by another ReplicatedSecret, so we can't do what you describe.

[burmanm]

We shouldn't have a case with Secret -> ReplicatedSecret -> Secret -> ReplicatedSecret -> Secret. Something's gone horrible wrong at that point and it's good that it fails.

[adejanovski]

Let's see if we can get away with a single ReplicatedSecret. Not sure the cluster label system will allow this, but worth a shot.

[Miles-Garnsey]

Let's see if we can get away with a single ReplicatedSecret. Not sure the cluster label system will allow this, but worth a shot.

I can't see any way to do this given that you want to replicate from MedusaConfig namespace -> K8ssandraCluster namespace -> DC namespace.

This functionality will go away once we finally remove the ability to have the MedusaConfig in a different namespace to the K8ssandraCluster, but as long as we maintain support for that functionality, we probably want to merge this to cover off a somewhat dangerous bug which I suspect can hit cluster performance pretty bad.

[Miles-Garnsey]

I guess the alternative here is to do something like add an annotation to the replica secret which notes which ReplicatedSecret created it (or use the OwnerReference). We could then filter out any secret which had been replicated once while still allowing these multi-hop replicas.

Let me know if you'd prefer that approach.