CheckRackLabels patching is ignored?

[burmanm]

What happened?

In this comment on PR: #698 (comment)

The StS labels were not changed in CheckRackLabels() for some reason. We need to investigate if we can replicate this or if this was just something Kubernetes opted not to do in this test run.

The unit tests have not caught this sort of behavior.

What did you expect to happen?

CheckRackLabels() should always update the StS ObjectMeta labels (not Spec labels) on every run.

How can we reproduce it (as minimally and precisely as possible)?

Not known.

cass-operator version

1.22.4

Kubernetes version

Method of installation

No response

Anything else we need to know?

No response

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-74

[burmanm]

This unit test seems to work fine for StS:

func TestCheckRackLabels(t *testing.T) {
	rc, _, cleanupMockScr := setupTest()
	defer cleanupMockScr()
	require := require.New(t)
	err := rc.CalculateRackInformation()
	require.NoError(err)

	desiredStatefulSet, err := newStatefulSetForCassandraDatacenter(
		nil,
		"default",
		rc.Datacenter,
		3)
	require.NoErrorf(err, "error occurred creating statefulset")

	desiredStatefulSet.Status.ReadyReplicas = *desiredStatefulSet.Spec.Replicas

	trackObjects := []runtime.Object{
		desiredStatefulSet,
		rc.Datacenter,
	}
	rc.Client = fake.NewClientBuilder().WithStatusSubresource(rc.Datacenter).WithRuntimeObjects(trackObjects...).Build()

	rc.statefulSets = []*appsv1.StatefulSet{desiredStatefulSet}

	res := rc.CheckRackLabels()
	require.Equal(result.Continue(), res, "Label updates should not cause errors")
	require.Subset(desiredStatefulSet.Labels, rc.Datacenter.GetRackLabels("default"))
	desiredStatefulSet.Labels[api.RackLabel] = "r1"
	require.NotSubset(desiredStatefulSet.Labels, rc.Datacenter.GetRackLabels("default"))

	res = rc.CheckRackLabels()
	require.Equal(result.Continue(), res, "Label updates should not cause errors")
	require.Subset(desiredStatefulSet.Labels, rc.Datacenter.GetRackLabels("default"))
}

[burmanm]

I can't replicate this behavior with manual testing either.

[Miles-Garnsey]

In manual testing, I do:

make create-kind-cluster cert-manager # (from k8ssandra-operator)
helm install --namespace cass-operator --create-namespace k8ssandra k8ssandra/cass-operator --set global.clusterScoped=true
kubectl apply -f tests/testdata/default-three-rack-three-node-dc-4x.yaml #(with some changes to the manifest so we have `datacenterName: DC&n_!!` and `size: 1`)
 helm upgrade --namespace cass-operator k8ssandra k8ssandra/cass-operator --reuse-values --set image.registry="michaelburman290" --set image.tag="v1.22.4-dev.a08e8eb-20241024"

I had to go manually edit the cass-operator deployment to get michaelburman290/cass-operator:v1.22.4-dev.a08e8eb-20241024 as image.

I then see:

template:
      metadata:
        creationTimestamp: null
        labels:
          app.kubernetes.io/created-by: cass-operator
          app.kubernetes.io/instance: cassandra-cluster1
          app.kubernetes.io/managed-by: cass-operator
          app.kubernetes.io/name: cassandra
          app.kubernetes.io/version: 4.1.6
          cassandra.datastax.com/cluster: cluster1
          cassandra.datastax.com/datacenter: DCn
          cassandra.datastax.com/node-state: Ready-to-Start
          cassandra.datastax.com/rack: default
...
metadata:
    annotations:
      cassandra.datastax.com/resource-hash: cnWZxVtbq2P3v/qFieMCgdJulBmPuFbG/d+kOF27N1Y=
    creationTimestamp: "2024-10-25T03:39:04Z"
    generation: 1
    labels:
      app.kubernetes.io/created-by: cass-operator
      app.kubernetes.io/instance: cassandra-cluster1
      app.kubernetes.io/managed-by: cass-operator
      app.kubernetes.io/name: cassandra
      app.kubernetes.io/version: 4.1.6
      cassandra.datastax.com/cluster: cluster1
      cassandra.datastax.com/datacenter: DCn
      cassandra.datastax.com/rack: default
...

I waited another 10 minutes just to be sure, eventually indeed I got:

metadata:
    annotations:
      cassandra.datastax.com/resource-hash: cnWZxVtbq2P3v/qFieMCgdJulBmPuFbG/d+kOF27N1Y=
    creationTimestamp: "2024-10-25T03:39:04Z"
    generation: 1
    labels:
      app.kubernetes.io/created-by: cass-operator
      app.kubernetes.io/instance: cassandra-cluster1
      app.kubernetes.io/managed-by: cass-operator
      app.kubernetes.io/name: cassandra
      app.kubernetes.io/version: 4.1.6
      cassandra.datastax.com/cluster: cluster1
      cassandra.datastax.com/datacenter: test-dc
      cassandra.datastax.com/rack: default
    name: cluster1-dcn-default-sts

So it looks like I just wasn't waiting long enough.

I do wonder whether the selector.matchLabels and template.metadata.labels fields will come back to bite us, as they never get updated (by design).

Is this a weird little bit of hidden complexity that we should try to iron out?

[burmanm]

I do wonder whether the selector.matchLabels and template.metadata.labels fields will come back to bite us, as they never get updated (by design).

They get updated once it's allowed. So once there's any change to CassandraDatacenter object or allowupdate-spec=once/always is set as annotation, we will update them.