slurm scheduler statistics

[martbhell]

http://giovannitorres.me/graphing-sdiag-with-graphite.html

Grab slurm statistics with https://github.com/PySlurm/pyslurm

The statistics from sdiag can be quite useful - we should make a collectd script which outputs the slurm statistics.

The output from sinfo/squeue can also be of interest (to show pending nodes/cores/memory).

[martbhell]

One could also perhaps use these: collectd/collectd#1198

[A1ve5]

ansible-role-xdmod: CSCfi/ansible-role-xdmod#1
It's running in io-grid

[martbhell]

Using xdmod is much better than using collectd/grafana, as one can drill down into the statistics. Shall we keep this open until we have an xdmod that is installed once per site or CSC has a central one?

[martbhell]

I couldn't find any of the "sdiag" statistics in xdmod - stats from the scheduler. For example how long it takes for the backfill scheduler to run, how deep it looks. Are they in there somewhere? Can we add them?

[martbhell]

Let's try the collectd way.