Cluster with external DC has auth issues: "Cannot achieve consistency level LOCAL_ONE"

[adejanovski]

When creating a new cluster using an external DC, the Cassandra pods will be provisioned, but the reaper pod is not provisioned, and auth does not work with the superuser account (nodetool, cqlsh ,etc.).

❯ kubectl get pods -n training-feq80wdm
NAME READY STATUS RESTARTS AGE
training-cluster-jedi-rack1-sts-0 2/2 Running 0 53m
training-cluster-jedi-rack2-sts-0 2/2 Running 0 53m
training-cluster-jedi-rack3-sts-0 2/2 Running 0 53m

❯ kubectl exec -it -n training-feq80wdm training-cluster-jedi-rack1-sts-0 -c cassandra -- /bin/bash
dse@training-cluster-jedi-rack1-sts-0:$ nodetool -u training-cluster-superuser -pw password status
nodetool: Failed to connect to '127.0.0.1:7199' - FailedLoginException: 'Failed to login. Please re-try.'.
dse@training-cluster-jedi-rack1-sts-0:$ cqlsh -u training-cluster-superuser -p password 10......
Connection error: ('Unable to connect to any servers', {'10.....:9042': AuthenticationFailed('Failed to authenticate to 10.40.2.14:9042: Error from server: code=0100 [Bad credentials] message="Failed to login. Please re-try."',)})
The DSE logs identify this as a problem, which will be repeatably written to the logs.

INFO [mainIOThread-9] 2024-03-20 19:19:39,558 RpcMethodDse68.java:138 - Failed to execute method NodeOps.createRole
java.lang.reflect.InvocationTargetException: null
...
Caused by: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_ONE
To workaround this, I had to alter the system_auth keyspace on the external DC to replicate to the MC DC. Afterward, auth works and the cluster finishes provisioning the reaper pod, alters non-user keyspaces RF, etc.

It appears that when we perform such an expansion, we wait for the CassandraDatacenter to be ready before updating the schema, but the state of the schema makes the role updates fail in cass-operator, preventing the cassdc to get to a ready state.
Updating the schema before the readiness wait could potentially unblock this case.