Question about RabbitMQ Memory Usage especially about "other_system"

[BBQigniter]

I'm currently testing setting up RabbitMQ-Clusters via the kubernetes-operator on Kubernetes 1.24. I tried to setup a small 3-node cluster and initially tried to set the resource request/limits as seen in the examples to 2G and 2 CPUs. But with this the pods where instantly OOMkilled.
I increased to 4G - pods now where ready - but after connecting to the management GUI I recognized that already a memory alert showed up on all three "nodes" because 1.6GiB is in use. After digging in I saw that "other_system" (Erlang?) is using most of the configured memory already with 1.5GiB.
So after configuring 6G as resource memory request/limit I now have an OK-looking cluster. But I wonder why the "other_system" is using that much without anything going on (there are no queues or whatever yet configured)

I have several other bare-metal clusters - with much older RabbitMQ (3.8) and Erlang (23.3) versions running, and they use much less memory even with load.

So is that now normal that with RabbintMQ 3.9/3.10 and Erlang 24 there is already much higher memory usage from the beginning on? Or do I have already some strange issue :D?

[mkuratczyk]

It's definitely not expected to require a few GB of RAM for an empty cluster. Can you provide your RabbitmqCluster YAML, rabbitmq-diagnostics observer screenshots, rabbitmq-diagnostics memory_breakdown output, logs and whatever else you can think of? :)

[BBQigniter]

Okay - interesting. I also tried it with replicas: 1 after deleting the pvs and data, so everything got recreated freshly. Again other_system was at 1.5G

The Kubernetes cluster consists of 3 Virtualbox VMs (2 master-nodes with worker role and 1 worker - I know not optimal but for testing it should be sufficient), with NAT (10.0.2.15) and host-only interface (192.168.56.111-113).

NAME      STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION            CONTAINER-RUNTIME
vkube-1   Ready    control-plane,worker   26h   v1.24.3   192.168.56.111   <none>        Fedora Linux 36 (Thirty Six)   5.18.11-200.fc36.x86_64   containerd://1.6.6
vkube-2   Ready    control-plane,worker   26h   v1.24.3   192.168.56.112   <none>        Fedora Linux 36 (Thirty Six)   5.18.11-200.fc36.x86_64   containerd://1.6.6
vkube-3   Ready    worker                 26h   v1.24.3   10.0.2.15        <none>        Fedora Linux 36 (Thirty Six)   5.18.11-200.fc36.x86_64   containerd://1.6.6   ### ooh - prolly this is an issue, will readd that node with a correct internal-ip asap

Just spotted a misconfiguration in the Kubernetes config, perhaps this causes something strange issue. Will correct that and add an reply asap if this was the fix. In the meantime, here is the yaml I use.

The yaml

# for creating the namespace if it's missing
apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq-buffer
---
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-buffer
  namespace: rabbitmq-buffer
spec:
  #image: rabbitmq:3.9.21-management
  #image: rabbitmq:3.10.2-management
  image: rabbitmq:3.10.6-management
  replicas: 3
  # some requests and limits
  resources:
    requests:
      cpu: 2
      memory: 6Gi
    limits:
      cpu: 2
      memory: 6Gi
  rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      # https://www.rabbitmq.com/memory.html#paging
      vm_memory_high_watermark_paging_ratio = 0.75
      vm_memory_high_watermark.relative = 0.4
      # https://www.rabbitmq.com/quorum-queues.html#resource-use - default 512 mebibytes - has no effect on other_system
      raft.wal_max_size_bytes = 67108864
      # https://www.rabbitmq.com/disk-alarms.html#configure
      disk_free_limit.relative = 1.0
      # set default user and password
      default_user=admin
      default_pass=123456
  # persistent volume config
  persistence:
    storageClassName: local-storage
    # lets see if this has effect
    storage: "5Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-buffer-management
  namespace: rabbitmq-buffer
spec:
  ports:
  - name: management
    port: 15672
    protocol: TCP
  selector:
    app.kubernetes.io/name: rabbitmq-buffer
  type: LoadBalancer
  loadBalancerIP: "192.168.56.121"

My persistent volumes are setup like this

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-001-rabbitmq-buffer-server-0
spec:
  # has no effect but needed
  capacity:
    storage: 5Gi
  accessModes:
  - ReadWriteOnce
  # Delete is not working in this case!
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    # the folder has to be created before applying this yaml
    path: /data/pv-001-rabbitmq-buffer-server-0
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - vkube-1
  claimRef:
    # we want that the first pod/es-node is always on this node
    # the pattern is: persistence-CLUSTERNAME-server-PODNUMBER
    name: persistence-rabbitmq-buffer-server-0
    namespace: rabbitmq-buffer

The kubectl get pv command's output

NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                  STORAGECLASS    REASON   AGE
pv-001-rabbitmq-buffer-server-0   5Gi        RWO            Retain           Bound    rabbitmq-buffer/persistence-rabbitmq-buffer-server-0   local-storage            22h
pv-002-rabbitmq-buffer-server-1   5Gi        RWO            Retain           Bound    rabbitmq-buffer/persistence-rabbitmq-buffer-server-1   local-storage            22h
pv-003-rabbitmq-buffer-server-2   5Gi        RWO            Retain           Bound    rabbitmq-buffer/persistence-rabbitmq-buffer-server-2   local-storage            22h

Inside of one of the pods:

The memory_breakdown

Reporting memory breakdown on node rabbit@rabbitmq-buffer-server-0.rabbitmq-buffer-nodes.rabbitmq-buffer...
other_system: 1.6244 gb (73.69%)
allocated_unused: 0.5119 gb (23.22%)
code: 0.0358 gb (1.63%)
other_proc: 0.0185 gb (0.84%)
plugins: 0.0052 gb (0.24%)
other_ets: 0.0034 gb (0.15%)
mgmt_db: 0.0022 gb (0.1%)
atom: 0.0015 gb (0.07%)
binary: 0.0011 gb (0.05%)
metrics: 2.0e-4 gb (0.01%)
mnesia: 1.0e-4 gb (0.01%)
connection_other: 0.0 gb (0.0%)
msg_index: 0.0 gb (0.0%)
quorum_ets: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
connection_readers: 0.0 gb (0.0%)
connection_writers: 0.0 gb (0.0%)
connection_channels: 0.0 gb (0.0%)
queue_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

Some observer output

HOME output
|Home(H)|Network(N)|System(S)|Ets(E)/Mnesia(M)|App(A)|Doc(D)|Plugin(P)recon:proc_count(memory, 54) Interval:5000ms     | 0Days 2:17:29    |
|Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit]                                            | 
|System     | Count/Limit           | System                    | Status                | Stat Info            | Size                     |
|Proc Count | 403/1048576           | Version                   | 24                    | Active Task          | 1                        | 
|Port Count | 14/134217727          | ps -o pcpu                | 5.2%                  | Context Switch       | 3457653                  |
|Atom Count | 47148/5000000         | ps -o pmem                | 21.0%                 | Reds(Total/SinceLast | 9287681699/50588         |
|Mem Type   | Size                  | Mem Type                  | Size                  | IO/GC:(5000ms)       | Total/Increments         |
|Total      | 1.5764 GB    | 100.0% | Binary                    | 997.8672 KB  | 00.06% | IO Output            | 197.5527 MB/227 B        | 
|Process    | 24.7820 MB   | 01.54% | Code                      | 34.1812 MB   | 02.12% | IO Input             | 58.7266 MB/69 B          | 
|Atom       | 1.3675 MB    | 00.08% | Port Parallelism (+spp)   | false                 | Gc Count             | 182489/0                 | 
|Ets        | 3.7719 MB    | 00.23% | RunQueue                  | 0                     | Gc Words Reclaimed   | 1046816814/0             |
|No | Pid        |     Memory   |Name or Initial Call                  |           Reductions| MsgQueue |Current Function                 |
|1  |<0.764.0>   |    1.8566 MB |worker_pool_worker:init/1             |             79166487| 0        |gen_server2:process_next_msg/1   |
|2  |<0.685.0>   |    1.1470 MB |delegate_management_1                 |              8243740| 0        |gen_server2:process_next_msg/1   |
|3  |<0.221.0>   |    1.0637 MB |rabbit                                |              2528127| 0        |application_master:loop_it/4     |
|4  |<0.44.0>    |  674.3359 KB |application_controller                |              2235403| 0        |gen_server:loop/7                |
|5  |<0.772.0>   |  449.2109 KB |prometheus_sup                        |                64488| 0        |gen_server:loop/7                |
|6  |<0.50.0>    |  416.4219 KB |code_server                           |              1623434| 0        |code_server:loop/1               |
|7  |<0.10.0>    |  416.4219 KB |erl_prim_loader                       |              8010541| 0        |erl_prim_loader:loop/3           |
|8  |<0.59.0>    |  277.9766 KB |file_server_2                         |               982994| 0        |gen_server:loop/7                |
|9  |<0.733.0>   |  277.8516 KB |application_master:start_it/4         |                84254| 0        |application_master:loop_it/4     |
|10 |<0.703.0>   |  225.0625 KB |node_coarse_metrics_metrics_collector |               550941| 0        |gen_server:loop/7                |
|11 |<0.680.0>   |  192.4219 KB |rabbit_mgmt_external_stats            |           8856241746| 0        |gen_server:loop/7                |

SYSTEMS output
|Home(H)|Network(N)|System(S)|Ets(E)/Mnesia(M)|App(A)|Doc(D)|Plugin(P)Interval: 5000ms                                 | 0Days 2:19:29    |
|System/Architecture    | State    | CPU's and Threads       | State   | Memory Usag | State                  | Statistics  | State       |
|System Version         | 24       | Logical CPU's           | 4       | Total       | 1.5756 GB     | 100.0% | ps -o pcpu  | 5.1%        |
|Erts Version           | 12.3.2.2 | Online Logical CPU's    | 4       | Processes   | 23.8925 MB    | 01.48% | ps -o pmem  | 21.1%       |
|Emulator Wordsize      | 8        | Available Logical CPU's | 4       | Atoms       | 1.3988 MB     | 00.09% | ps -o rss   | 1.6393 GB   |
|Process Wordsize       | 8        | Schedulers              | 4       | Binaries    | 1.0060 MB     | 00.06% | ps -o vsz   | 4.5295 GB   |
|Smp Support            | true     | Online schedulers       | 4       | Code        | 34.1812 MB    | 02.12% | Total IOIn  | 59.2462 MB  |
|Thread Support         | true     | Available schedulers    | 4       | Ets         | 3.7830 MB     | 00.23% | Total IOOut | 201.0767 MB |
|compiled for           | x86_64-pc-linux-gnu                                                                                             |
|Allocator Type   | Current Mbcs     | Max Mbcs         | Current Sbcs     | Max Sbcs         | Current SbcsToMbcs  | Max SbcsToMbcs      |
|binary_alloc     | 1.5679 KB        | 2.0017 KB        | 0 B              | 631.9609 KB      | 0.0000              | 0.0000              |
|driver_alloc     | 0.2622 KB        | 5.8827 KB        | 0 B              | 0 B              | 0.0000              | 0.0000              |
|eheap_alloc      | 8.8477 KB        | 7.2684 KB        | 609.9167 KB      | 1.4498 MB        | 0.0066              | 0.0344              |
|ets_alloc        | 0.6745 KB        | 0.6720 KB        | 0 B              | 0 B              | 0.0000              | 0.0000              |
|fix_alloc        | 0.2340 KB        | 0.2294 KB        | 0 B              | 0 B              | 0.0000              | 0.0000              |
|ll_alloc         | 20.7685 KB       | 20.7671 KB       | 0 B              | 0 B              | 0.0000              | 0.0000              |
|sl_alloc         | 0.1328 KB        | 0.1533 KB        | 0 B              | 1.0240 MB        | 0.0000              | 0.0004              |
|std_alloc        | 0.4252 KB        | 0.5855 KB        | 0 B              | 0 B              | 0.0000              | 0.0000              |
|temp_alloc       | 0 B              | 28.7454 KB       | 0 B              | 1.2584 MB        | 0                   | 0.1538              |
|Instance | Hits       | Calls       | Hit Rate                                                                                           |
|00       | 326        | 585         | ||||||||||||||||||||||||||||||||||||||||||||||||||                                        | 55.73% |
|01       | 15659      | 25719       | |||||||||||||||||||||||||||||||||||||||||||||||||||||||                                   | 60.88% |
|02       | 2888       | 5258        | |||||||||||||||||||||||||||||||||||||||||||||||||                                         | 54.93% |
|03       | 41         | 75          | |||||||||||||||||||||||||||||||||||||||||||||||||                                         | 54.67% |
|04       | 49         | 57          | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||            | 85.96% |
|q(quit)                                                                                                                                  |

And here are some screenshots from the management UI

[BBQigniter]

hmm, fixing the node's internal IP didn't help

[mkuratczyk]

I'm afraid I don't know what can be causing this. I can recommend setting up Prometheus/Grafana and the Erlang Memory dashboard: https://grafana.com/grafana/dashboards/11350. Perhaps this will give us some insights.

[BBQigniter]

In the meantime I completely did a new setup of my small k8s-cluster with bridged network interfaces to make sure they are not causing this issue, but it didn't help. And here you are - ll_alloc uses most memory

It is because of the super-huge "File (1073741816 available) and Socket (966367545 available) descriptors"-limits - I found this interesting thread http://erlang.org/pipermail/erlang-questions/2016-May/089335.html after googling for "erlang ll_alloc memory usage".

But as new k8s-clusters do not use Docker anymore - how to set it for pods or containerd properly? In the systemd-unit file of containerd there is only a note

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.

Anyway for testing I now set LimitNOFILE=64000 via systemctl edit containerd and poof - initial memory usage is reduced to 132MiB. I would rather be able to set the ulimit somehow in the "rabbitmq-cluster.yaml"

[mkuratczyk]

Thanks for investigating. Can you tell me where these limits (1073741816/966367545) are set?

[BBQigniter]

well, like I wrote they are somehow "forwarded" by containerd. In the default containerd systemd unit file you find for example:

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

With systemctl edit containerd I overide the LimitNOFILE value now - as the comments above state, this is not a good solution.

I think I would have to add somehow a command to the initContainer to set the ulimit correctly and leave the default values in the systemd unit-file. Something like

command: ["sh", "-c", "ulimit -n 65536"] 

[ikavgo]

Maybe related: #959

[ikavgo]

also docker-library/rabbitmq#545

[BBQigniter]

yes just wanted to post this :D and was preparing this post

more googling led me to docker-library/rabbitmq#545 and further to this already closed issue #959

I now applied the "workaround" mentioned in comment #959 (comment)

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: override-erl-max-ports
spec:
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq
              env:
              - name: ERL_MAX_PORTS
                value: "50000"

and set the systemd unit-file overrides back to default values. Now memory usage is in normal boundaries

The strange thing is, I already tried to configure ERL_MAX_PORTS via the normal operator's supplied envConfig-option which had no effect

...
  rabbitmq:
    envConfig: |
      ERL_MAX_PORTS=65536
...

also tried

...
  rabbitmq:
    envConfig: |
      ERL_MAX_PORTS="65536"
...

Looking inside the container I couldn't find the environment variable being set.

So for the moment I can live with the workaround. The full kind: RabbitmqCluster-yaml file now looks like this

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-buffer
  namespace: rabbitmq-buffer
spec:
  image: rabbitmq:3.10.6-management
  # workaround for Erlang high memory usage!
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq
              env:
              - name: ERL_MAX_PORTS
                value: "65535"
  replicas: 3
  # some requests and limits
  resources:
    requests:
      cpu: 2
      memory: 6Gi
    limits:
      cpu: 2
      memory: 6Gi
  rabbitmq:
    # this seems to have NO effect
    #envConfig: |
    #  ERL_MAX_PORTS="65536"
    additionalConfig: |
      cluster_partition_handling = pause_minority
      # https://www.rabbitmq.com/memory.html#paging
      vm_memory_high_watermark_paging_ratio = 0.75
      vm_memory_high_watermark.relative = 0.4
      # https://www.rabbitmq.com/quorum-queues.html#resource-use - default 512 mebibytes
      raft.wal_max_size_bytes = 67108864
      # https://www.rabbitmq.com/disk-alarms.html#configure
      disk_free_limit.relative = 1.0
      # set default user and password
      default_user=admin
      default_pass=123456
  # persistent volume config
  persistence:
    storageClassName: local-storage
    # lets see if this has effect
    storage: "5Gi"