<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and (-webkit-min-device-pixel-ratio: 2) {
#email-body { min-width: 30em !important; }
#email-page { padding: 8px !important; }
#email-banner { padding: 8px 8px 0 8px !important; }
#email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
#email-fields { padding: 0 8px 8px 8px !important; }
#email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#f0f0f0;color:#000000;width:100%;">
<tr valign="top">
<td id="email-page" style="padding:16px !important;">
<table align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;">
<tr valign="top">
<td bgcolor="#ffffff" style="background-color:#ffffff;color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img src="https://shibboleth.net/images/shib_240x83.png" alt="" style="vertical-align:top;" /></td>
</tr><tr valign="top">
<td id="email-banner" style="padding:32px 32px 0 32px;">
<table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="width:100%;">
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
<img id="email-avatar" src="https://issues.shibboleth.net/jira/secure/useravatar?avatarId=10202" alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px 16px 0;" />
<div id="email-action" style="padding: 0 0 8px 0;font-size:12px;line-height:18px;">
<a class="user-hover" rel="bkoehmstedt@ucmerced.edu" id="email_bkoehmstedt@ucmerced.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=bkoehmstedt%40ucmerced.edu" style="color:#326ca6;">Brian Koehmstedt</a>
created <img src="https://issues.shibboleth.net/jira/images/icons/issuetypes/bug.png" height="16" width="16" border="0" align="absmiddle" alt="Bug"> <a style='color:#326ca6;text-decoration:none;' href='https://issues.shibboleth.net/jira/browse/SC-195'>SC-195</a>
</div>
<div id="email-summary" style="font-size:16px;line-height:20px;padding:2px 0 16px 0;">
<a style='color:#326ca6;text-decoration:none;' href='https://issues.shibboleth.net/jira/browse/SC-195'><strong>OpenJDK-only Rhino memory leak</strong></a>
</div>
</td>
</tr>
</table>
</td>
</tr>
<tr valign="top">
<td id="email-fields" style="padding:0 32px 32px 32px;">
<table border="0" cellpadding="0" cellspacing="0" style="padding:0;text-align:left;width:100%;" width="100%">
<tr valign="top">
<td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
<td>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Issue Type:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<img src="https://issues.shibboleth.net/jira/images/icons/issuetypes/bug.png" height="16" width="16" border="0" align="absmiddle" alt="Bug"> Bug
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Assignee:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<a class="user-hover" rel="cantor.2@osu.edu" id="email_cantor.2@osu.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=cantor.2%40osu.edu" style="color:#326ca6;">Scott Cantor</a>
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Created:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
18/Sep/14 5:34 PM
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Description:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<p style='margin-top:0;margin-bottom:10px;'>Relevant shibusers thread about this where this was discussed in more detail:<br/>
<a href="http://marc.info/?t=141092427700002&r=1&w=2" class="external-link">http://marc.info/?t=141092427700002&r=1&w=2</a><br/>
"Memory leak hunting - Rhino script woes leading to stale metadata not being garbage collected"</p>
<p style='margin-top:0;margin-bottom:10px;'>OpenJDK seems to ship with a Rhino engine that differs in some way with the Oracle JDK Rhino, and this difference leads to a memory leak when running Shibboleth with OpenJDK and a configuration that uses Rhino Javascript (at least OpenJDK on RedHat Linux 6.5). Once this leak was duplicated in test, Oracle JDK was tried, and the leak was not observed.</p>
<p style='margin-top:0;margin-bottom:10px;'>When this memory leak was observed, both filtering and resolver scripts were in use. I am not sure if the leak would be observed with resolver-only scripts, but at a minimum the leak was traced back to the Rhino engines that run filter scripts.</p>
<p style='margin-top:0;margin-bottom:10px;'>By analyzing heap dumps, the leak seems to stem from the following:</p>
<ul class="alternate" type="square">
<li>In ScriptMatchFunctor.java, in getScriptContext(), there is:<br/>
scriptContext.setAttribute("filterContext", ...)</li>
<li>In the heap dump, there are instances of a Rhino ClassCache that gets created. The purpose of this cache seems to cache the reflection of methods and member fields of the class belonging to filterContext. The result of this caching seems to also hold a reference to the instance of the filterContext as well (for a reason I am not sure). From there references are held to stale EntitiesDescriptorImpl. These EntitiesDescriptorImpls can build up and cause the memory leak as multiple scripts are run and the metadata is updated over time.</li>
</ul>
<p style='margin-top:0;margin-bottom:10px;'>(One attribute / one script will only result in one stale filterContext. But if you use filters for multiple attributes, then these multiple, different filterContexts, all with potentially different refs to EntitiesDescriptorImpl, can end up getting cached.)</p>
<p style='margin-top:0;margin-bottom:10px;'>One potential fix, as referenced in a few other posts on the Internet, is to try setting scriptContext.setOptimizationLevel(-1) to prevent class caching in Rhino (this is specific to Rhino and not relevant to other JSR scripting engines...plus Rhino is going away in JDK 8 anyway.)</p>
<p style='margin-top:0;margin-bottom:10px;'>I have not tried to rebuild Shib with this fix yet so I cannot yet verify that it actually works, but I am willing to try out a candidate build with my test cases to verify. This could hurt script performance, but Scott points out it could be a configuration item (assuming the fix works).</p>
<p style='margin-top:0;margin-bottom:10px;'>From the email thread, this is how I described how I duplicated this in test:<br/>
(RedHat v6.5, OpenJDK u55-b13):</p>
<ul class="alternate" type="square">
<li>Wrote a quick and dirty script to every minute update the<br/>
<ds:Reference URI="..."> in a metadata file copied from the Incommon file.<br/>
(May not be necessary to randomize URI. Changing the last modified time <br/>
stamp may be sufficient. Not sure.)<br/>
This was so that the test Shib IdP will recognize a new metadata file.</li>
<li>Run a web server that serves up this metadata file</li>
<li>Configure the FileBackedHTTPMetadataProvider in relying-party.xml to<br/>
download this changed file every minute.</li>
<li>Create a dummy attribute (or just use an existing one) in LDAP</li>
<li>Edit attribute-filter.xml and add 5 to 10<br/>
<afp:AttributeFilterPolicy><afp:PolicyRequirementRule/><afp:AttributeRule/></afp:AttributeFilterPolicy> <br/>
sections that reference the dummy attribute and for each one, provide a <br/>
different matching value. (Example below.)</li>
<li>Then test by setting the dummy attribute value to, say, 1 and clearing<br/>
all cookies, and logging into a test SP. (Also, of course, release the <br/>
attribute to the SP that you have in afp:AttributeRule.)</li>
<li>Allow the FileBackedHTTPMetadataProvider to download a new version of<br/>
the metadata (it should do this every minute.)</li>
<li>Grab the PID of your IDP JVM</li>
<li>Do jmap -histo:live <PID> | grep EntitiesDescriptorImpl and note the<br/>
count.</li>
<li>Then repeat the above 4 steps, except increment your dummy value to be<br/>
something different now so that a new <afp:AttributeFilterPolicy> is <br/>
triggered (matched) on your next login.</li>
</ul>
<p style='margin-top:0;margin-bottom:10px;'>As you repeat those steps, it accumulates cached <br/>
ShibbolethFilteringContexts and EntitiesDescriptorImpl objects in the heap.</p>
<p style='margin-top:0;margin-bottom:10px;'>Example attribute-filter.xml config using a dummy attribute with <br/>
incrementing values:</p>
<p style='margin-top:0;margin-bottom:10px;'> <!-- run the filtering script for givenName if MatchingAttribute==1 --><br/>
<afp:AttributeFilterPolicy><br/>
<afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"<br/>
attributeID="MatchingAttribute"<br/>
value="1"/><br/>
<afp:AttributeRule attributeID="givenName"><br/>
<afp:DenyValueRule xsi:type="basic:Script"><br/>
<basic:Script><br/>
<![CDATA[<br/>
<SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS><br/>
]]><br/>
</basic:Script><br/>
</afp:DenyValueRule><br/>
</afp:AttributeRule><br/>
</afp:AttributeFilterPolicy></p>
<p style='margin-top:0;margin-bottom:10px;'> <!-- run the filtering script for givenName if MatchingAttribute==2 --><br/>
<afp:AttributeFilterPolicy><br/>
<afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"<br/>
attributeID="MatchingAttribute"<br/>
value="2"/><br/>
<afp:AttributeRule attributeID="givenName"><br/>
<afp:DenyValueRule xsi:type="basic:Script"><br/>
<basic:Script><br/>
<![CDATA[<br/>
<SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS><br/>
]]><br/>
</basic:Script><br/>
</afp:DenyValueRule><br/>
</afp:AttributeRule><br/>
</afp:AttributeFilterPolicy></p>
</td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Environment:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<p style='margin-top:0;margin-bottom:10px;'>RedHat 6.5, OpenJDK, IdP 2.4.0<br/>
OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)</p>
</td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Project:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<a style="color:#326ca6;" href="https://issues.shibboleth.net/jira/browse/SC">Shibboleth Common - Java</a>
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Priority:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<img src="https://issues.shibboleth.net/jira/images/icons/priorities/minor.png" height="16" width="16" border="0" align="absmiddle" alt="Minor"> Minor
</td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Reporter:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<a class="user-hover" rel="bkoehmstedt@ucmerced.edu" id="email_bkoehmstedt@ucmerced.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=bkoehmstedt%40ucmerced.edu" style="color:#326ca6;">Brian Koehmstedt</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td><!-- End #email-page -->
</tr>
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: 0 16px 16px 16px;text-align:center;">
This message is automatically generated by JIRA.<br />
If you think it was sent incorrectly, please contact your JIRA administrators<br />
For more information on JIRA, see: <a style='color:#326ca6;' href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
</td>
</tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->