Memory leak hunting - Rhino script woes leading to stale metadata not being garbage collected

[Mark Boyce]

Ted,

Thanks!  Although I would be exceedingly cautious of allowing Java to update via YUM or any other automated process.  Could end up with "unintended consequences".

Cheers,

Mark

Senior Identity Management Analyst
Universiity of California, Office of the President


-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Ted Fisher
Sent: Thursday, September 18, 2014 4:44 AM
To: Shib Users
Subject: RE: Memory leak hunting - Rhino script woes leading to stale metadata not being garbage collected


Off topic here, but you can use Oracle JDK 1.7.0_67 Linux x64 and still have Redhat package management keep it up to date, etc.  Redhat has additional repositories including one called "Oracle Java for RHEL".  Subscribe to that repo and you can load directly from it with yum and it will get updated when you do yum updates.  We use nothing but Oracle/Sun JVM on all RHEL systems and all are managed this way.

Ted F. Fisher
Information Technology Services
BGSU, Bowling Green, Ohio

-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Brian Koehmstedt
Sent: Thursday, September 18, 2014 12:01 AM
To: Shib Users
Subject: Re: Memory leak hunting - Rhino script woes leading to stale metadata not being garbage collected

I was able to duplicate this issue in test with RedHat/OpenJDK by utilizing the following trick:

- Wrote a quick and dirty script to every minute update the <ds:Reference URI="..."> in a metadata file copied from the Incommon file.
(May not be necessary to randomize URI.  Changing the last modified time stamp may be sufficient.  Not sure.) This was so that the test Shib IdP will recognize a new metadata file.
- Run a web server that serves up this metadata file
- Configure the FileBackedHTTPMetadataProvider in relying-party.xml to download this changed file every minute.
- Create a dummy attribute (or just use an existing one) in LDAP
- Edit attribute-filter.xml and add 5 to 10 <afp:AttributeFilterPolicy><afp:PolicyRequirementRule/><afp:AttributeRule/></afp:AttributeFilterPolicy>
sections that reference the dummy attribute and for each one, provide a different matching value.  (Example below.)
- Then test by setting the dummy attribute value to, say, 1 and clearing all cookies, and logging into a test SP.  (Also, of course, release the attribute to the SP that you have in afp:AttributeRule.)
- Allow the FileBackedHTTPMetadataProvider to download a new version of the metadata (it should do this every minute.)
- Grab the PID of your IDP JVM
- Do jmap -histo:live <PID> | grep EntitiesDescriptorImpl and note the count.
- Then repeat the above 4 steps, except increment your dummy value to be something different now so that a new <afp:AttributeFilterPolicy> is triggered (matched) on your next login.

As you repeat those steps, it accumulates cached ShibbolethFilteringContexts and EntitiesDescriptorImpl objects in the heap.

Once I was able to duplicate the memory leak in test like this, it became easy to test with a different JVM.

So I tested with Oracle JDK 1.7.0_67 Linux x64 instead of the OpenJDK that ships with RedHat 6.5.

The jmap -histo:live shows that the EntitiesDescriptorImpl count does
*not* grow with this JVM (and no stale ShibbolethFilteringContexts).

Now I'm ready to blame OpenJDK.  :)

I'll still need to verify by deploying a new Oracle JVM to production and confirming the memory leak is gone, but based on the above testing, it looks likely that switching to Oracle JDK and avoiding the JVM that ships with RedHat will solve the issue.

Too bad because it would be nice to use the package management system on RedHat for software updates like the JVM.  That was one of the reasons why we migrated to RedHat in the first place.

A couple side notes:

- I found a few things on the web that suggests using
scriptContext.setOptimizationLevel(-1) will prevent Rhino from using the ClassCache.  (Perhaps the Oracle JDK has this nonoptimization level by default while OpenJDK simply has it set to optimize by default?)
   - One potential fix to try for OpenJDK deployments is add
scriptContext.setOptimizationLevel(-1) to the Shibboleth code to see if it avoids the ClassCaching.  And if so, I wonder if it's worth considering adding to the code.  (Is this nonoptimization level the default in Oracle JDK?)
- I saw postings on the web that indicate Rhino has been removed from JDK 8 in favor of a "Nashorn."  I wonder what the implications of this are for anybody who is migrating shibboleth to Java 8 and who has existing Javascript scripts in their Shibboleth configs?

Example attribute-filter.xml config using a dummy attribute with incrementing values:

     <!-- run the filtering script for givenName if MatchingAttribute==1 -->
     <afp:AttributeFilterPolicy>
         <afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"
                                    attributeID="MatchingAttribute"
                                    value="1"/>
         <afp:AttributeRule attributeID="givenName">
             <afp:DenyValueRule xsi:type="basic:Script">
               <basic:Script>
                 <![CDATA[
                   <SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS>
                 ]]>
               </basic:Script>
             </afp:DenyValueRule>
         </afp:AttributeRule>
     </afp:AttributeFilterPolicy>

     <!-- run the filtering script for givenName if MatchingAttribute==2 -->
     <afp:AttributeFilterPolicy>
         <afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"
                                    attributeID="MatchingAttribute"
                                    value="2"/>
         <afp:AttributeRule attributeID="givenName">
             <afp:DenyValueRule xsi:type="basic:Script">
               <basic:Script>
                 <![CDATA[
                   <SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS>
                 ]]>
               </basic:Script>
             </afp:DenyValueRule>
         </afp:AttributeRule>
     </afp:AttributeFilterPolicy>

--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
-- 
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net