Cannot add nodes to EKS cluster after the k8ssandra-operator installed in this cluster and delete all nodes

[xkxrxm]

What happened?

We installed the k8ssandra-operator in our EKS cluster(for test), and we delete all nodes after work every day. But when we add nodes to EKS cluster next day, the nodes cannot be ready.

Did you expect to see something different?

Nodes be ready

How to reproduce it (as minimally and precisely as possible):

1. Install the k8ssandra-operator:

helm install k8ssandra-operator k8ssandra/k8ssandra-operator -n k8ssandra-operator --create-namespace

2. Delete all nodes and wait for Scheduling until

➜  kubectl get nodes
No resources found

3. Add nodes back, but nodes cannot be ready

➜  kubectl get nodes
NAME                                               STATUS     ROLES    AGE    VERSION
ip-****.ap-northeast-1.compute.internal   NotReady   <none>   9m7s   v1.28.3-eks-e71965b

Environment

• K8ssandra Operator version:

  v1.11.0

• Kubernetes version information:

Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4-eks-8cb36c9

• Kubernetes cluster kind:

  eks

• Manifests:

insert manifests relevant to the issue

• K8ssandra Operator Logs:

insert K8ssandra Operator logs relevant to the issue here

Anything else we need to know?:

After we uninstalled the k8ssandra-operator, We can add nodes normally.

[adejanovski]

Hi @xkxrxm,

I think this could be related to this issue. The Mutating Webhook with k8ssandra-operator down could prevent the creation of pods that will be necessary to bring the node to a ready state (such as coredns, etc...).

I'll release a patch version of the operator today with the fix and update this ticket so that you can test again.

[xkxrxm]

Thanks for your quick response.

[xkxrxm]

@adejanovski Thanks for your quick response again, I tested on version 1.12.0-20240116080917-413f04f3 just now, it run normally

[adejanovski]

Thanks for testing so quickly!

I've released v1.11.0 with the fix.