Alerting for high memory usage?

[sebhoss]

We ran into high memory watermark today in one of our clusters and realized that we do not have any alerts for that situation. I looked around and it seems this was once discussed in #1006 but closed.

My question: Is the formula rabbitmq_resident_memory_limit_bytes - rabbitmq_process_resident_memory_bytes < 0 the recommended approach to monitor high memory usage? The numbers look good to me, but I wanted to make sure, that we are not missing anything.

Any help highly appreciated ❤️

[Zerpet]

You can benefit of our Grafana dashboards if you consume metrics using Prometheus and let Grafana fire the alerts for you.

The formula you shared is correct to calculate the number of bytes before you hit the memory alarm. Note that rabbitmq_resident_memory_limit_bytes is the "maximum" number of bytes utilised before rabbit fires a memory alarm. This metric does not correlate, nor match the memory limit defined in your resource limits (in the kubernetes manifest).

Answering your question, yes, that's a good method to monitor for memory usage 🙂 The above definition may or may not match your definition of "high memory usage", specially given that the default high memory watermark value is 40% of the resource limit e.g. 400 Mi if you assigned 1Gi as memory limit.

[sebhoss]

that is an awesome answer - thanks a lot!