shibd Memory Usage

[Andy Bennett]

Hi,

>> Thanks for the info. My restarted shibd is already at 388MiB Virtual and
>> 148MiB Resident.
> 
> I just restarted it on shibboleth.net, which loads a lot of metadata.
> After a single load to start up, it's at 312M. So, basically, that's just
> how it is until the federations switch to on-demand metadata. DOM loads
> are expensive, and unlike on Windows, the memory doesn't come back. That
> same memory footprint on Windows will shrink down to under 100M because
> the DOMs dropping frees up a ton of heap.
> 
> Under most conditions, it shouldn't grow enormously because each DOM load
> reuses most of the earlier heap blocks, but there's always some growth
> over time.

On Linux with glibc large allocations will be entirely allocated with
mmap() and unmap()'d on free. Small allocations will use sbrk() and
never get given back to the kernel.

Are you allocating all the memory up front for each metadata ingest or
does each xml node end up with a "small" allocation? If it ends up using
exclusively small allocations and getting a bit fragmented I can imagine
it having these memory usage symptoms.


>> I've not seen any other long running processes eat into swap like this
>> but maybe it's a function of the frequently reallocated metadata
>> structures?
> 
> Yes. I forced Xerces to tune that down a bit, but it's still bad. And I
> don't even know what Xerces you're using. I think the worst numbers are
> far back in history, but I'd be leery of anything before 3.0.

It looks like I've got something in the 3.1 series:

$ dpkg -l |grep xerc
ii  libxerces-c3.1                      3.1.1-1+b1
validating XML parser library for C++



Regards,
@ndy

-- 
andyjpb at knodium.com
http://www.knodium.com/