How to explain more quorum queues will consume more memory resources in case of producing same amount of message volumes

[xixiangzouyibian]

I have 3 nodes rabbitmq cluster to declare quorum queues.
Test message size is 10K, rabbitmq node resource limit is 2 CPU and 4GB memory.

Tested against cluster with different scenarios (the value means percentage of 4GB memory consuming):

10 quorum queues, each queue has 1 producer and 0 consumer, producing total 2G message volume: 
Node1   Node2    Node3
11.72%  17.22%   16.72%

50 quorum queues, each queue has 1 producer and 0 consumer, producing total 2G message volume: 
Node1   Node2    Node3
24.92%  25.07%   15.85%

200 quorum queues, each queue has 1 producer and 0 consumer, producing total 2G message volume: 
Node1   Node2    Node3
33.72%  24.42%   32.86%  ---> almost reaches high watermark

So I want to know what causes the quorum queue consume more memory with the same amount of messages generated ?
And the memory consumption seems to grow at such a high rate.

[mkuratczyk]

This should answer at least some questions: https://www.rabbitmq.com/quorum-queues.html#resource-use

Moreover:

1. Ra log WAL (use by quorum queues) is initially randomized to prevent a roll over on all nodes at the same time. Hence, the usage on all nodes in a cluster will intentionally differ (this is likely why you see eg. 24.42% on node2 and 32.86% on node3 in the last case)
2. Each queue is an Erlang process and therefore consumes some memory. Therefore, memory usage with 10 empty quorum queues will be different than memory usage with 200 empty queues (adding messages introduces additional variability, but the difference will be there even without messages)

[xixiangzouyibian]

Thanks for your response.
Let me have a brief summary:

This should answer at least some questions: https://www.rabbitmq.com/quorum-queues.html#resource-use

According to the article, in-memory index of message will consume some memory resources. But for my case, because of same total message volume, I suppose this kind of memory cost is the same for all of my scenarios, and it is cheap as 32 bytes per message.

Ra log WAL (use by quorum queues) is initially randomized to prevent a roll over on all nodes at the same time. Hence, the usage on all nodes in a cluster will intentionally differ (this is likely why you see eg. 24.42% on node2 and 32.86% on node3 in the last case)

WAL cost is one of reasons. Can I understand like more quorum queues will generate more logs ?

Each queue is an Erlang process and therefore consumes some memory. Therefore, memory usage with 10 empty quorum queues will be different than memory usage with 200 empty queues (adding messages introduces additional variability, but the difference will be there even without messages)

Erlang process is one of other reasons that more quorum queues result in more process created and more memory resources consumed.

By the way, according to my understandings, more quorum queues produce more replicas, is this another reason to consume more memory resources based on this behavior ?

[mkuratczyk]

The total memory used for in-memory indexes can be the same, but it matters whether it's 1 Erlang process using all of it or many processes using just a little. This will lead to a different memory allocation and garbage collection patterns.

All quorum queues on a single node share the write-ahead-log. I guess there might be some small differences but generally speaking, the total memory used be the WAL will be the same regardless of the number of queues.

I'm not sure what you mean by producing replicas. Each quorum queue is a single Erlang process on every node that has a member of that queue (leader or follower). So 1 queue in a 3-node cluster means 3 processes (1 per node) and 100 queues in a cluster means 300 processes. There are caveats like not fully replicated queues, larger clusters (a quorum queue won't have more than 3 members by default) and so on, but most commonly: 1 QQ means 1 process per node.

[xixiangzouyibian]

Ok. So can I say this: the main reasons for the increase in memeory consumption is that the erlang processes have become more numerous ?

[mkuratczyk]

It's definitely a contributing factor. But please do not conclude from this that you should use 1 queue for the best performance. :) Like everything, it's a tradeoff between the business needs (ordering of messages), parallelism (a single Erlang process can only use 1 CPU at a given time) and other factors such as memory usage.

You are asking interesting questions but there are no trivial answers here. It's a combination of RabbitMQ design, Erlang runtime and configuration, operating system memory management and exact timing of many events.

[xixiangzouyibian]

Thanks mkuratczyk. I'm very appreciate for our conversation to talk about this question.
I hope this is an open post to attract more friends to brainstorm ideas.