the relationship between metadata size and heap size

[Simon Lundström]

On Mon, 2015-09-14 at 08:35:29 -0400, Tom Scavo wrote:
> On Mon, Sep 14, 2015 at 4:29 AM, Simon Lundström <simlu at su.se> wrote:
> > On Sun, 2015-09-13 at 17:39:20 +0000, Cantor, Scott wrote:
> >> On 9/12/15, 10:22 AM, "users on behalf of Tom Scavo" <users-bounces at shibboleth.net on behalf of trscavo at gmail.com> wrote:
> >> >
> >> >What makes this worse is that the InCommon metadata aggregate will
> >> >double in size overnight (literally) once we start importing eduGAIN
> >> >metadata. If I knew what to tell deployers to prepare for this event,
> >> >I would. Is there some JVM configuration that is known to work with
> >> >large aggregates of size 32MB and beyond?
> >>
> >> […]I don't know why people are having problems (I'm not, with 2.4.x on Java 7/8) and I don't know why it's not logging a simple error if its out of memory, but people are on so many different versions at this point.
> >
> > Tom, how often do you reload/refresh the metadata? When we refreshed the
> > metadata every 10 minutes we ran out of memory every month or so. Our
> > federation (SWAMID) metadata is 20MB atm.
> 
> InCommon participants are advised to attempt a refresh every hour but
> surely some don't and just take the default. This would explain why
> some deployments exhibit the issues while others don't.
> 
> > This disappeared when we changed it to the default (4h).
> 
> Very interesting. Another piece of the puzzle :-)
> 
> Thanks Simon.

Np! I forgot that I had memory usage graphs before and after (not that
long after, but you get the idea at least = )
<https://stockholmuniversity.box.com/shared/static/s6hmjxbh0h7le73tyh7j7ee32rgq0gep.png>

The IDP is restarted just after 08:00 because of OOME and then at 15:30
with the 4h.

BR,
- Simon