shibd massive memory growth and CPU usage

[Stockley, Jonathan]

We are definitely not using prefork:
$ ls -l /usr/sbin/apache2
lrwxrwxrwx 1 root root 33 2014-06-24 00:51 /usr/sbin/apache2 -> ../lib/apache2/mpm-worker/apache2
Apache version is 2.2.14
I had already previously set the stack size to 128K using ulimit in the startup script just to be sure.

Don't know if this is relavant but I've been looking at the apache error logs and at the time of the problem there are about 100 messages like this:
[Thu Jul 10 23:45:01 2014] [error] [client 10.8.46.43] Failure receiving response to remoted message (default/SAML2/POST)., referer: https://adfs.xxx.yyy.com/adfs/ls/?SAMLRequest=jZJLb8IwEIT%2FSuR74rwQwiJIKRyKRFsEtIdeKsfZgCXHTr1OS%2F998%2BiDXlCvu7Mz40%2BeI69Vw%2FLWnfQOXltA551rpZENi4y0VjPDUSLTvAZkTrB9frdhcRCyxhpnhFHEyxHBOmn00mhsa7B7sG9SwONuk5GTcw0ySitpO3ujQchAyqNQpi0DxYsAahEIU9P9SRaFUeBOAaKhfU5Mtw%2F7A%2FFW3aXUvI%2F4NeRlhYE%2BSuwd4Ox6s2FIFVLirVcZeSmKqpxBwqsqqsRMxJN4ytNJIkoepUkEaSdDbGGt0XHtMhKHUeqHUz8KD3HC4phF02fibb9eeiN1KfXxOpZiFCG7PRy2%2Ftj%2FCSwO3TsBWcx7uGwIthe4r9vyb8Zk8W%2Bi%2BEPUx2ZOL2LHDg2773LWq61RUnx4uVLmfWmBO8hIROhiPPn7Pxaf&RelayState=ss%3Amem%3A7c9cc55f619df39ed56cc027e29aa43646b4c13d5ed5ce392e77e925a670cce2

Is there any debugging I can turn on to get more info?

Looking at the transaction log file I see that the NameIdentifier is always the same. I'll try turning off maintainReverseIndex and see if that helps.

Jo

-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Cantor, Scott
Sent: Thursday, July 10, 2014 5:58 PM
To: Shib Users
Subject: Re: shibd massive memory growth and CPU usage

On 7/10/14, 8:30 PM, "Stockley, Jonathan" <jonathan.stockley at emc.com>
wrote:

>All goes well for a few days then they run the tests again and I see 
>about 100+ shibd threads and VIRT and RES both go up to 3.5G and total 
>CPU goes to about 700%, ie 7 of the 8 CPUs are pegged.

And are you using Apache 1.3 or prefork mode? If so, either stop, or change the thread stack size.

The other issue is with load testing if the same NameID is sent over and over. The SP will not handle that well, and it's not a real world condition. You can work around that while testing by turning maintainReverseIndex off in the SessionCache, and the bottleneck will be avoided.

But seeing 100+ threads is a strong indicator of prefork.

> 
>It seems to me that sessions (that will never be used again) are 
>building up far faster than they expire and get cleaned out.

Well, load testing will do that, but they will go away. I doubt that's the issue.

>I¹m thinking that maybe I need to set Session timeout to something 
>small, around 10 and reduce the SessionCache inprocTimeout to 10, 
>cleanupInterval to 60 and then set the memory cleanupInterval to 60 as 
>well.
> 
>Am I in the right ball park? Any ideas?

Those values are in seconds, so no, that would not be useful.

>I¹m also being asked ³How will we handle 1000s of simultaneous logins?²

Simultaneous sessions and logins are very different things, but answer #1 is to not use prefork.

-- Scott

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net