The RabbitMQ cluster does not provide high availability (HA).

[sdwan444]

Describe the bug

Hello, we are experiencing an issue with our RabbitMQ cluster. We have three nodes in the cluster using the Bitnami operator. When one node goes down, we encounter errors, and some of our queues turn red (increasing by 0).

Expected behavior
When node is failing the messages should keep going.

Screenshots

If applicable, add screenshots to help explain your problem.

Version and environment information

• RabbitMQ: [e.g. 3.12.12]
• RabbitMQ Cluster Operator:(rabbitmq-cluster-operator:2.7.0-debian-11-r3)
• Kubernetes: [e.g. 1.28.0]
• Cloud provider or hardware configuration: k8s with spot instance.

[lukebakken]

Hello, thanks for using RabbitMQ.

We need more information before we can assist you.

• Are you running in AWS? https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html
• How exactly are you configuring the RabbitMQ cluster operator in your k8s environment? Please provide your FULL configuration yaml.
• How are you stopping one node in this cluster?
• Please provide your full log files. NOT screenshots. ATTACH your compressed log files to this discussion.

[sdwan444]

Hey, Thx for the quick response, unfortunately i can't share the log file publicly.

The node stops because of spot switch with evict. and it does a "drain" action.

The yml is:

Yaml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  spec:
  delayStartSeconds: 30
  image: docker.io/bitnami/rabbitmq:3.12.12-debian-11-r10
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers: []
            topologySpreadConstraints:
              - labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: rabbitmq
                maxSkew: 10
                topologyKey: topology.kubernetes.io/zone
                whenUnsatisfiable: ScheduleAnyway
  persistence:
    storage: 20Gi
    storageClassName: gp3
  rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      vm_memory_high_watermark_paging_ratio = 0.99
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 10000
      vm_memory_high_watermark.relative = 0.8
    additionalPlugins:
      - rabbitmq_amqp1_0
      - rabbitmq_management
      - rabbitmq_event_exchange
      - rabbitmq_web_dispatch
      - rabbitmq_management_agent
      - rabbitmq_stomp
      - rabbitmq_web_stomp
      - rabbitmq_web_mqtt
      - rabbitmq_mqtt
  replicas: 3
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: '1'
      memory: 3Gi
  secretBackend:
    externalSecret: {}
  service:
    type: ClusterIP
  terminationGracePeriodSeconds: 604800

[lukebakken]

Without the logs we will be unable to continue assisting. Our first suggestion is, don't use spot instances.

[sdwan444]

there is any log part that is going to help? the complete file can not go public.
So if im not using spot when the node will go down im going to face the same problem?

[michaelklishin]

@sdwan444 you must not use spot instances with Raft-based features. They assume reasonably stable identity of nodes, not that they are completely volatile.

[sdwan444]

So what im going to do if one of my nodes is going down?

[mkuratczyk]

Set up a test environment without spot instances, try to reproduce the problem and if you can - report it with the logs included (they would be test instance logs, so I hope shareable). Otherwise you are basically asking us to do this work.

[sdwan444]

@lukebakken, @michaelklishin
The log file.
logs.log

[mkuratczyk]

This is the same thing you shared before. It's an important part, but only a part. We need logs from all nodes and not just the error - there could have been other important messages before this happened.

[sdwan444]

there is any best practices regarding leader election? or node switching?