shibd Memory Usage

[Andy Bennett]

Hi,

>> We tend to find that it likes to use a lot of RAM. I always assumed it
>> was just a function of the number of transactions. However, I've just
>> looked on one of our machines that doesn't actually do any Shibboleth
>> stuff but has the same configuration installed as our other machines.
>> It's been sitting there for 600 days or so and the logs show that all
>> it's been doing is downloading the metadata feed: it hasn't processed
>> any transactions. That shib was using some 700MiB of RAM.
> 
> That's not all that shocking given that Linux, unlike Windows, does not
> free heap. This is known as "broken", but it dates from the standard Unix
> philosophy that threads are evil and process churn isn't.

That's an interesting and good point!


> That said, you can certainly check and make sure you're not running Apache
> prefork to rule that out as a source of RAM spikes.

We're not running Apache at all.

We use the FastCGI binaries but all the memory usage is by shibd.


>> Is that the amount that would be expected (given the size of the
>> ukfederation metadata of a few thousand entries) or is something amiss?
> 
> I don't really know what it uses on a routine basis, but all it takes is a
> spike, because once it's allocated, it ain't coming back. It's virtual of
> course, and empty heap will just end up on disk, but it's still going to
> be allocated.

Yes: we see it eat into swap but it doesn't degrade performance. That
would correlate with unused pages being swapped out and never touched again.


> There were a few small leaks fixed in 2.5.x, IIRC.






Regards,
@ndy

-- 
andyjpb at knodium.com
http://www.knodium.com/