RabbitMQ Operator and OpenELB Annotations

[rkazak07]

Hello,

I'm running a RabbitMQ cluster with the RabbitMQ operator and exposing it as a LoadBalancer. I'm not experiencing any issues with access or application performance. However, I constantly see the following output in the OpenEBS logs:

2024-03-10T20:17:34.192339124+03:00 {"level":"info","ts":1710091054.192074,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
{"level":"info","ts":1710091054.2178903,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
2024-03-10T20:17:34.218452387+03:00 {"level":"info","ts":1710091054.2180402,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
{"level":"error","ts":1710091054.2340772,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"LBController","request":"rabbitmq-cluster/rabbitmqcluster","error":"Operation cannot be fulfilled on services \"rabbitmqcluster\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90"}
2024-03-10T20:17:35.234993335+03:00 {"level":"info","ts":1710091055.234604,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
2024-03-10T20:17:35.235106817+03:00 {"level":"info","ts":1710091055.2347758,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
2024-03-10T20:17:35.266078826+03:00 {"level":"info","ts":171
0091055.2657273,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
{"level":"info","ts":1710091055.2659442,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
2024-03-10T20:17:35.286362044+03:00 {"level":"info","ts":1710091055.2860568,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
2024-03-10T20:17:35.286672285+03:00 {"level":"info","ts":1710091055.286245,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
2024-03-10T20:17:35.341497745+03:00 {"level":"info","ts":1710091055.3412046,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
2024-03-10T20:17:35.341702347+03:00 {"level":"info","ts":1710091055.3413813,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
2024-03-10T20:17:35.365004038+03:00 {"level":"info","ts":1710091055.3645945,"msg":"setup openelb service","service":"rabbitmq-cluster/rabbitmqcluster"}
2024-03-10T20:17:35.365133937+03:00 {"level":"info","ts":1710091055.3649418,"logger":"IPAM","msg":"unAssignIP","args":{"Key":"rabbitmq-cluster/rabbitmqcluster","Addr":"","Eip":"opn-el-eip","Protocol":"layer2","Unalloc":false},"peek":true,"result":{"Addr":"10.36.100.23","Eip":"opn-el-eip","Protocol":"layer2","Sp":{}},"err":null}
2024-03-10T20:17:35.384218848+03:00 {"level":"error","ts":1710091055.383832,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"LBController","request":"rabbitmq-cluster/rabbitmqcluster","error":"Operation cannot be fulfilled

Here is my RabbitMQ YAML configuration:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
 name: rabbitmqcluster
spec:
 image: bitnami/rabbitmq:3.12.13-debian-12-r2
 replicas: 3
 resources:
   limits:
     cpu: 2
     memory: 4Gi
 rabbitmq:
   additionalPlugins:
     - rabbitmq_management
     - rabbitmq_peer_discovery_k8s
   additionalConfig: |
     default_user = rabbit-admin
     default_pass = Pasword
   advancedConfig: ""


 persistence:
   storageClassName: nfs-client
   storage: "20Gi"

 service:
   annotations:
     lb.kubesphere.io/v1alpha1: openelb
     protocol.openelb.kubesphere.io/v1alpha1: layer2
     eip.openelb.kubesphere.io/v1alpha2: opn-el-eip
   type: LoadBalancer

Kubernetes: v1.27.10
RabbitMQ version: bitnami/rabbitmq:3.12.13-debian-12-r2
Operator version: 3.20.1
OpenELB version: 0.5.0
OS: Debian11

Could you please help me understand why I'm seeing these errors in the OpenEBS logs? Is there any misconfiguration or issue with the annotations in my RabbitMQ YAML?

Thank you.

[lukebakken]

This appears to be the relevant message:

Operation cannot be fulfilled on services \"rabbitmqcluster\": the object has been modified; please apply your changes to the latest version and try again

I am not a k8s expert, maybe @mkuratczyk or @Zerpet can assist you.

Google may be of help:

https://www.google.com/search?q=%22please+apply+your+changes+to+the+latest+version+and+try+again%22

[mkuratczyk]

My best guess is that there are multiple operators "competing" to manage a resource - whenever one of them touches the object, the other one is notified of the object modification and "touches it" as well and they go like in circles like that. If we get full reproduction steps, starting with an empty cluster, we can take a look since it could be an issue in resource ownership or something.