Rabbitmq Operator deploys rabbitmq cluster as 2 clusters #1472
-
I deployed 3-node rabbitmq cluster by rabbitmq operator, which specified replicas as 3. But it finally formed 2 rabbitmq clusters and they didn't recognize each other (empty list in network partitions). The detail of observations:
The split-brain behavior I understand is generate a cluster of 3 nodes at the beginning, but the cluster is split into 2 due to network partiion failure. However, in my case, the deployment from the beginning is two clusters. Only one of clusters provide the services, and the other one appears idle. How can I do for this case or how to mitigate this senario, and is it possible to restore the cluster as one ? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 14 replies
-
The phenomenon occurs frequently in our pipeline deployment, not sure if it is a rabbitmq or operator issue.
|
Beta Was this translation helpful? Give feedback.
-
@mkuratczyk I updated rabbitmq to 3.11.23 since this is the version supported by latest rabbitmq operator: https://artifacthub.io/packages/helm/bitnami/rabbitmq-cluster-operator
The issue still persists (2-node formed a cluster and 1-node formed the other cluster). |
Beta Was this translation helpful? Give feedback.
-
@mkuratczyk @Zerpet Hi Team, After nearly 4 days observation, the statistical sample about the issue of cluster formation split is summarized as below ("1" means problematic deployment, "0" means non-problematic deployment): Almost 10% of the problem probability is very high for a product. Please feel free to find the attachments for further analysis. |
Beta Was this translation helpful? Give feedback.
-
@mkuratczyk We updated rabbitmq version from bitnami 3.11.23-debian-11-r22 to official 3.12.7-management and no issue found anymore after 3 days monitoring, totally 67 times deployment. We regard the issue has been really resolved. Thanks for your great support. |
Beta Was this translation helpful? Give feedback.
As I said above, you can simply set
image: rabbitmq:3.12.7-management
to deploy 3.12 (with therabbitmq
image in this case). There's nothing to "adapt"