Look into memory issues on the web nodes

[simong]

This is almost definitely due to the Shibboleth SP. I assume it's because it needs to keep some user sessions and/or idp metadata into memory. We should look into why this is happening.

This is a low priority issue though.

[davidoae]

It looks like in production shib is holding 1.5G of data (not code) of data in memory. Much of this is swapped out. This probably slows all shib requests, which isn't serious now but may well become more noticeable as more users use shib.
Also the disk cache is only ~17m which is pathetic and likely slows down disk requests. The small cache is due to processes actually using physical memory so the kernel doesn't have much headroom to play with.
As the server only has 1G of physical memory my recommendation is to have double the amount that shib requires, ie we should upgrade so we have 3+G of physical memory.

[simong]

Before we double the amount of memory, is there any indication of why it's using that much memory? From what I remember, the graphs seemed to show a steady growth of memory usage? Are we not releasing session information or might there be a memory leak? I know we're a patch version behind, are there any performance related fixes in there?

[davidoae]

In September there was a large increase is memory use, probably when you implemented shib.
At that point, straight away, the server no longer had enough memory to be nicely efficient.
Since then the use has increased slowly by a couple of hundred MB.
We're using shib 2.5.3, the latest version is 2.5.4.
I checked the release notes. There are some minor bug fixes and 1 security update to prevent DOS on certain types of malformed requests. The updates are not critical, although we should probably do the upgrade. I looked through their JIRA for that release and I didn't spot anything memory leak or performance related.
I haven't found any strong references to shib memory leaks in the version we're using.
We should upgrade anyway but I don't believe it's super urgent. Requires checking some related software that shib uses, like openssh.
From what I've read Shib doesn't release memory back to the OS, however the size should only grow as the number of shib sessions increases.
The memory numbers for our shib seem roughly in line with other people's reported use.
So maybe we should upgrade shib and increase the memory :)

[mrvisser]

So maybe we should upgrade shib and increase the memory :)

+1 if it will improve authentication time. Would be interesting to see if the memory just continues to climb to the new memory limit.

[davidoae]

There no evidence that it's expanding to use available memory, there's
plenty of swap remaining.
It should give a slight improvement to auth time. It should also make nginx
slightly faster too as more content will be cached by kernel.

On 4 June 2015 at 12:51, Branden Visser [email protected] wrote:

So maybe we should upgrade shib and increase the memory :)

+1 if it will improve authentication time. Would be interesting to see if
the memory just continues to climb to the new memory limit.

—
Reply to this email directly or view it on GitHub
#124 (comment)
.