Monitoring or managing IDP memory usage

[Wessel, Keith]

All,

Looking for advice on monitoring IDP heap space utilization.

Last week, our 2GB heap space got exhausted and left our IDP unable to resolve metadata for SPs from InCommon or eduGAIN. I bumped it up from 2 GB to 3 GB, and things have been fine ever since.

However, I wanted to monitor and prevent future problems before they happen. So, I wrote a quick Nagios plugin that grabs the used and max memory from the status handler.

And according to the plugin, as soon as our Nagios admin turned it on, we were out of heap space. The status page does, in fact, report that we're using approx.. 2964 MB of a max 2964 MB.

First question: is the maximum memory the max memory available to the IDP, or is it the max memory that has been used by the IDP within the current running instance?

If it's the max that has been used, what would be a safe number to use in my Nagios plugin when calculating heap space utilization? Can I assume a certain percentage of my 3 GB is used for system overhead and the rest is available to the IDP? Any guess how much overhead that might be?

If that max memory is really the max available to the IDP, sounds like I either need to do some garbage collection more often or increase my heap space again. I'm familiar with garbage collection back until Tomcat and IDP v2, but is there something I should be doing in V3 with Jetty?

If all of this fails, I'll just bump up my head space. I do find it hard to believe that we're using all 3 GB, though.

Looking forward to per-entity metadata queries!

Keith