The Dreaded PermGen Error

[Joshua Riffle]

Scott:

Jetty sounds interesting. I'll take a look down that path. A lot of us
don't view Jetty as a 'production' servlet system but that's probably just
engrained culture.

Sean:

I have a sneaking suspicion that my 'memory leak' only occurs when I
Stop/Start Shibboleth IDP in order to apply configuration changes. Reboot
seems to prevent it altogether so that means there's no proof that Tomcat
itself isn't the culprit and in fact the tomcat log files are very
ambiguous:

Nov 99, 2013 99:99:99 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: The web application [] created a ThreadLocal with key of type
[org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value
[org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1 at 7513edd0]) and
a value of type [byte[]] (value [[B at 252ff85f]) but failed to remove it when
the web application was stopped. This is very likely to create a memory
leak.

Note that it does actually give a name for the web application that it is
detecting a memory leak within, but this is followed by Shibboleth's
idp-process.log file entry:

15:44:08.384 - ERROR
[edu.internet2.middleware.shibboleth.common.profile.ProfileRequestDispatcherServlet:
89] - Error occurred while processing request
java.lang.OutOfMemoryError: PermGen space
15:44:08.420 - WARN
[edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:147] -
No login context available, unable to return to authentication engine



Joshua Riffle
Software Engineer
*Azusa Pacific University*


On Mon, Nov 11, 2013 at 8:57 AM, Flannery, Sean <sean.flannery at jwt.com>wrote:

> I do not run the IDP and am not an expert on that, but perhaps I can help
> you with the java part of your question
>
>
> 1. Increasing your MaxPermGenSize will, yes, only delay the invietable
> crash * if you have a class loader leak * but if you don't have a leak it
> could very well fix your problem.  Are you sure you have a leak?  I doubt
> Shibboleth has a native leak.  Have you tried a higher value?  Personally,
> I would increase your -XX:MaxPermSize option and see if it fixes the issue.
>
>
> 2. If increasing your PermGenSize does not help and you really have a
> class loader leaak- they are tricky to fix and the only real solution is
> to find it in a dev environment and fix it, i.e. there isn't a good way
> (that I know of) to be proactive about it.
>
> This website details how to trouble shoot such problems if you feel you do
> have a class loader leak:
>
> http://frankkieviet.blogspot.ca/2006/10/classloader-leaks-dreaded-permgen-s
> pace.html
>
>
>
>
> From:  Joshua Riffle <jriffle at apu.edu>
> Reply-To:  Shib Users <users at shibboleth.net>
> Date:  Monday, November 11, 2013 10:23 AM
> To:  users <users at shibboleth.net>
> Subject:  The Dreaded PermGen Error
>
>
> We have been getting this error from time to time with our Shibboleth
> Identity Provider in Tomcat 6 and before you reply to this thread...
>
>
> 1) Yes I know this is caused by a memory leak and can be 'mitigated' by
> increasing the max permgen size:
>
> https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepare
>  <
> https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepa
> re>
>
>
> 2) Or better yet possibly resolved in Tomcat 7:
> http://wiki.apache.org/tomcat/MemoryLeakProtection
> <http://wiki.apache.org/tomcat/MemoryLeakProtection>
>
>
> -- What I don't know and would LOVE input on is:
>
>
> 1) How do you detect or prevent the evil PermGen error beforehand? My
> understanding is increasing the MaxPermGenSize is only delaying the
> inevitable memory leak build up.
>
>
> 2) Better yet: Best practices? What have you already done in your
> architecture to avoid or defeat this kind of issue?
>
> Joshua RiffleSoftware Engineer
> Azusa Pacific University
>
> --
> To unsubscribe from this list send an email to
> users-unsubscribe at shibboleth.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20131111/b7ccc897/attachment.html