Shibboleth SP getting progressively slower

[Robert Egglestone]

Hi Everyone,

Under normal circumstances we see logins taking between 0.5s and 1s. Within the last month or so we've been seeing our login times getting slower at a rate of roughly 1 second every day. When we restart the Shibboleth SP, the login time go back to normal, and then start rising again.

The shibd memory usage is approx 200 to 300Mb, and the CPU usage isn't excessive.

There are graphs showing the login times from our test system below. Where there are gaps, this is usually due to the IdP being offline. Where there are sudden spikes short load tests are usually being run.
http://robert.enarc.auckland.ac.nz/~regg002/shibsp-slow-2012jan/month.png
http://robert.enarc.auckland.ac.nz/~regg002/shibsp-slow-2012jan/week.png

The degradation doesn't appear to get worse after performing a large number of logins.

The SP we're running is shibboleth-2.4.3-2.2.el5 on Red Hat Enterprise release 5.6, using Apache 2.2.17.
We have 2 servers in test, 3 in prod, and all show the same issue. We're not currently using odbc.
Prod servers have 3Gb of ram, test has 1Gb, they don't seem to be swapping.

Apache is setup as a reverse proxy with about 100 sites and 50 applications behind it, of which maybe 30 use Shibboleth.

Most of the login requests are against an IdP running Shibboleth 2.2.0.

Regards,
Robert Egglestone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20120119/156b2f91/attachment-0001.html