<div dir="ltr"><div><div><div><div>Scott:<br><br></div>Jetty sounds interesting. I'll take a look down that path. A lot of us don't view Jetty as a 'production' servlet system but that's probably just engrained culture.<br>
<br></div>Sean:<br><br></div>I have a sneaking suspicion that my 'memory leak' only occurs when I Stop/Start Shibboleth IDP in order to apply configuration changes. Reboot seems to prevent it altogether so that means there's no proof that Tomcat itself isn't the culprit and in fact the tomcat log files are very ambiguous:<br>
<br><div class="" title="Hint: double-click to select code"><div class=""><code class="">Nov </code><code class="">99</code><code class="">, </code><code class="">2013</code> <code class="">99</code><code class="">:</code><code class="">99</code><code class="">:99</code> <code class="">PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap</code></div>
<div class=""><code class="">SEVERE:
The web application [] created a ThreadLocal with key of type
[org.apache.xml.security.utils.UnsyncByteArrayOutputStream$</code><code class="">1</code><code class="">] (value [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$</code><code class="">1</code><code class="">@7513edd0</code><code class="">]) and a value of type [</code><code class="">byte</code><code class="">[]] (value [[B</code><code class="">@252ff85f</code><code class="">]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.</code></div>
</div><br></div><div>Note that it does actually give a name for the web application that it is detecting a memory leak within, but this is followed by Shibboleth's idp-process.log file entry:<br><div class="" title="Hint: double-click to select code">
<div class=""><code class=""><br>15</code><code class="">:</code><code class="">44</code><code class="">:</code><code class="">08.384</code> <code class="">- ERROR [edu.internet2.middleware.shibboleth.common.profile.ProfileRequestDispatcherServlet:</code><code class="">89</code><code class="">] - Error occurred </code><code class="">while</code> <code class="">processing request</code></div>
<div class=""><code class="">java.lang.OutOfMemoryError: PermGen space</code></div><div class=""><code class="">15</code><code class="">:</code><code class="">44</code><code class="">:</code><code class="">08.420</code> <code class="">- WARN [edu.internet2.middleware.shibboleth.idp.authn.AuthenticationEngine:</code><code class="">147</code><code class="">] - No login context available, unable to </code><code class="">return</code> <code class="">to authentication engine</code></div>
</div><br></div></div><div class="gmail_extra"><br clear="all"><div><font face="tahoma, sans-serif"><div><font face="tahoma, sans-serif"><br></font></div>Joshua Riffle</font><div><font face="tahoma, sans-serif">Software Engineer<br>
</font><div><font color="#CC0000" face="tahoma, sans-serif"><b>Azusa Pacific University</b></font></div></div></div>
<br><br><div class="gmail_quote">On Mon, Nov 11, 2013 at 8:57 AM, Flannery, Sean <span dir="ltr"><<a href="mailto:sean.flannery@jwt.com" target="_blank">sean.flannery@jwt.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I do not run the IDP and am not an expert on that, but perhaps I can help<br>
you with the java part of your question<br>
<br>
<br>
1. Increasing your MaxPermGenSize will, yes, only delay the invietable<br>
crash * if you have a class loader leak * but if you don't have a leak it<br>
could very well fix your problem. Are you sure you have a leak? I doubt<br>
Shibboleth has a native leak. Have you tried a higher value? Personally,<br>
I would increase your -XX:MaxPermSize option and see if it fixes the issue.<br>
<br>
<br>
2. If increasing your PermGenSize does not help and you really have a<br>
class loader leaak- they are tricky to fix and the only real solution is<br>
to find it in a dev environment and fix it, i.e. there isn't a good way<br>
(that I know of) to be proactive about it.<br>
<br>
This website details how to trouble shoot such problems if you feel you do<br>
have a class loader leak:<br>
<br>
<a href="http://frankkieviet.blogspot.ca/2006/10/classloader-leaks-dreaded-permgen-s
pace.html" target="_blank">http://frankkieviet.blogspot.ca/2006/10/classloader-leaks-dreaded-permgen-s<br>
pace.html</a><br>
<br>
<br>
<br>
<br>
From: Joshua Riffle <<a href="mailto:jriffle@apu.edu">jriffle@apu.edu</a>><br>
Reply-To: Shib Users <<a href="mailto:users@shibboleth.net">users@shibboleth.net</a>><br>
Date: Monday, November 11, 2013 10:23 AM<br>
To: users <<a href="mailto:users@shibboleth.net">users@shibboleth.net</a>><br>
Subject: The Dreaded PermGen Error<br>
<div class="im"><br>
<br>
We have been getting this error from time to time with our Shibboleth<br>
Identity Provider in Tomcat 6 and before you reply to this thread...<br>
<br>
<br>
1) Yes I know this is caused by a memory leak and can be 'mitigated' by<br>
increasing the max permgen size:<br>
<br>
<a href="https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepare" target="_blank">https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepare</a><br>
</div> <<a href="https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepa" target="_blank">https://wiki.shibboleth.net/confluence/display/SHIB2/IdPApacheTomcatPrepa</a><br>
re><br>
<div class="im HOEnZb"><br>
<br>
2) Or better yet possibly resolved in Tomcat 7:<br>
<a href="http://wiki.apache.org/tomcat/MemoryLeakProtection" target="_blank">http://wiki.apache.org/tomcat/MemoryLeakProtection</a><br>
<<a href="http://wiki.apache.org/tomcat/MemoryLeakProtection" target="_blank">http://wiki.apache.org/tomcat/MemoryLeakProtection</a>><br>
<br>
<br>
-- What I don't know and would LOVE input on is:<br>
<br>
<br>
1) How do you detect or prevent the evil PermGen error beforehand? My<br>
understanding is increasing the MaxPermGenSize is only delaying the<br>
inevitable memory leak build up.<br>
<br>
<br>
2) Better yet: Best practices? What have you already done in your<br>
architecture to avoid or defeat this kind of issue?<br>
<br>
Joshua RiffleSoftware Engineer<br>
Azusa Pacific University<br>
<br>
</div><div class="HOEnZb"><div class="h5">--<br>
To unsubscribe from this list send an email to <a href="mailto:users-unsubscribe@shibboleth.net">users-unsubscribe@shibboleth.net</a><br>
</div></div></blockquote></div><br></div>