the relationship between metadata size and heap size

[Cantor, Scott]

On 9/12/15, 10:22 AM, "users on behalf of Tom Scavo" <users-bounces at shibboleth.net on behalf of trscavo at gmail.com> wrote:
>
>What makes this worse is that the InCommon metadata aggregate will
>double in size overnight (literally) once we start importing eduGAIN
>metadata. If I knew what to tell deployers to prepare for this event,
>I would. Is there some JVM configuration that is known to work with
>large aggregates of size 32MB and beyond?

There's no formula, if that's what you're asking. I don't know why people are having problems (I'm not, with 2.4.x on Java 7/8) and I don't know why it's not logging a simple error if its out of memory, but people are on so many different versions at this point.

If you don't have at least a 1G heap allocated in 2015, I would say that's a good start. 1.5G if you can do it. 

But FWIW, mine's only 768M and I have multiple metadata sources that are >1M loaded on top of InCommon's. I don't know if that will handle the doubling in size, probably not.

>Every time I mention per-entity metadata and dynamic metadata query to
>someone, they quote the familiar adage that Shibboleth can handle much
>larger aggregates than we have today.

It can, but not with good performance. Even with a large heap, the signature check will become more and more inefficient. There are no free signature libraries capable of good performance on large files, and the SP's library is much worse than Java's is. It will become untenable within a few years of growth I would guess.

> It's time to stop fooling
>ourselves. This is Yet Another Reason to start taking per-entity
>metadata seriously, especially for IdPs.

I wish I knew what to do about that. We wrote the code, nobody wants to test it. I tend to think discovery is certainly one reason.

-- Scott