Shibboleth SP getting progressively slower

[Cantor, Scott]

> Under normal circumstances we see logins taking between 0.5s and 1s.
> Within the last month or so we've been seeing our login times getting slower
> at a rate of roughly 1 second every day. When we restart the Shibboleth SP,
> the login time go back to normal, and then start rising again.

What kind of load? We handle 100,000+ logins per day across some of our clusters, and there's nothing like that.

There's only one bottleneck I know of in the SP that is visible when you load test using a common NameID. If you send it thousands of sessions wth a common ID, there's a chokepoint in the code that doesn't handle that scenario because it's not really realisitic and I didn't handle it well.

Otherwise there aren't many real chokepoints unless your session count builds up very high. If you have reasonable timeouts set to purge them routinely, they wouldn't ever build up enough to matter.
 
> The shibd memory usage is approx 200 to 300Mb, and the CPU usage isn't
> excessive.

What about Apache?

> Apache is setup as a reverse proxy with about 100 sites and 50 applications
> behind it, of which maybe 30 use Shibboleth.

Worker MPM or "it's 2012 and we still think threads are voodoo" MPM? Use worker if you're not already.

-- Scott