shibd Memory Usage

[Cantor, Scott]

On 9/26/14, 12:42 PM, "Andy Bennett" <andyjpb at knodium.com> wrote:
>
>Are you allocating all the memory up front for each metadata ingest or
>does each xml node end up with a "small" allocation? If it ends up using
>exclusively small allocations and getting a bit fragmented I can imagine
>it having these memory usage symptoms.

Xerces suballocates out of a smallish malloc'd block and when that runs
out it mallocs a block twice as big, and keeps doubling the size until it
stops needing more. For metadata files, this grows quickly and then has
enough. For runtime messages, it needs to start small so it doesn't blow
up memory fast just because it's processing lots of requests. It used to
start at too large a size (64K, IIRC) so it would just get ridiculous.
It's down to 8K per initial allocation now because I gave them a patch
years ago.

TL;DR, there's a suballocator inside the DOM implementation and all the
memory for the parser comes from that suballocator. It does not allocate
chunks large enough to ever return any memmory.

-- Scott