Is memory watermark too low?

[GBwandersoncamargo]

Hi everyone 👋

Today a cluster was denying the publishing of messages because of memory limits.

Checking default configurations, is memory watermark too low?

Suppose we have configured resources limits with 100 units of memory.

RabbitMQ Operator will configure total_memory_available_override_value with 80% of this value to reserve 20% to Erlang.

So, RabbitMQ will be configured with 80 units of memory.

https://www.rabbitmq.com/kubernetes/operator/using-operator.html#resource-reqs

After that, RabbitMQ will configure vm_memory_high_watermark.relative equal to 40%. So, RabbitMQ will throttle publishers if memory is above 32 units of memory.

From the beggining, we lost 68 units of memory 😢

https://www.rabbitmq.com/memory.html#threshold

Should I increase watermark, in example, to 80%? Is this secure? In this case, we have some in-memory queues configured.

[mkuratczyk]

The 0.4 default is indeed very conservative and I'm pretty sure we'll revisit it at some point (on the rabbitmq-server inside, I don't mean overriding the default in the Operator). Therefore, changing this limit is definitely an option. Given how many ways there are to use RabbitMQ, with different versions, queue types, queue lengths and other factors, picking a good default is very hard and can certainly be adjusted on a per-environment basis where these details are known.

What do you mean by "in-memory queues" by the way? If you mean non-lazy, be advised that in 3.12, we'll make all queues lazy basically (which should reduce memory usage, especially if you switch to classic queues v2).