shibd massive memory growth and CPU usage

[Stockley, Jonathan]

Hi,
We're running Shibboleth SP 2.5.2 on linux host with 8 CPU and 4G RAM and I'm seeing the following behavior:

After startup all is fine and I see a handful of shibd threads, memory is at 165m VIRT 20m RES according to top command, CPU is negligible.

When our QE folks run their tests, for each test they authenticate anew so I see a new session created in shibd.log
New sessions are occurring about 2-4 per second.
All goes well for a few days then they run the tests again and I see about 100+ shibd threads and VIRT and RES both go up to 3.5G and total CPU goes to about 700%, ie 7 of the 8 CPUs are pegged.

Trying to authenticate take about 10 minutes under these conditions.
Restarting shibd clears the problem and everything settles down for a few more days.

I have session lifetime set to 3600 and timeout to 600, relayState is ss:mem

It seems to me that sessions (that will never be used again) are building up far faster than they expire and get cleaned out.
I've been looking at the docs for SessionCache and the memory StorageService but am not quite sure which values to tweak.

I'm thinking that maybe I need to set Session timeout to something small, around 10 and reduce the SessionCache inprocTimeout to 10, cleanupInterval to 60 and then set the memory cleanupInterval to 60 as well.

Am I in the right ball park? Any ideas?
I'm also being asked "How will we handle 1000s of simultaneous logins?"

Thanks,
Jo


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20140710/53647b3a/attachment.html