Memory leak hunting - Rhino script woes leading to stale metadata not being garbage collected
C R
publist.cr at gmail.com
Wed Oct 1 03:44:23 EDT 2014
For what it's worth, we (KU Leuven University) recently migrated our
production IdP install to Oracle JDK 7 from Redhat's OpenJDK 7. We
experienced problems where Tomcat wasn't reachable from Apache resulting in
many 503 errors. We experienced this twice in a week with very high
load/connections (start academic years).
With Oracle's JDK we haven't seen the problem yet.
Claudio
2014-09-18 17:57 GMT+02:00 Mark Boyce <Mark.Boyce at ucop.edu>:
> Ted,
>
> Thanks! Although I would be exceedingly cautious of allowing Java to
> update via YUM or any other automated process. Could end up with
> "unintended consequences".
>
> Cheers,
>
> Mark
>
> Senior Identity Management Analyst
> Universiity of California, Office of the President
>
>
> -----Original Message-----
> From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net]
> On Behalf Of Ted Fisher
> Sent: Thursday, September 18, 2014 4:44 AM
> To: Shib Users
> Subject: RE: Memory leak hunting - Rhino script woes leading to stale
> metadata not being garbage collected
>
>
> Off topic here, but you can use Oracle JDK 1.7.0_67 Linux x64 and still
> have Redhat package management keep it up to date, etc. Redhat has
> additional repositories including one called "Oracle Java for RHEL".
> Subscribe to that repo and you can load directly from it with yum and it
> will get updated when you do yum updates. We use nothing but Oracle/Sun
> JVM on all RHEL systems and all are managed this way.
>
> Ted F. Fisher
> Information Technology Services
> BGSU, Bowling Green, Ohio
>
> -----Original Message-----
> From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net]
> On Behalf Of Brian Koehmstedt
> Sent: Thursday, September 18, 2014 12:01 AM
> To: Shib Users
> Subject: Re: Memory leak hunting - Rhino script woes leading to stale
> metadata not being garbage collected
>
> I was able to duplicate this issue in test with RedHat/OpenJDK by
> utilizing the following trick:
>
> - Wrote a quick and dirty script to every minute update the <ds:Reference
> URI="..."> in a metadata file copied from the Incommon file.
> (May not be necessary to randomize URI. Changing the last modified time
> stamp may be sufficient. Not sure.) This was so that the test Shib IdP
> will recognize a new metadata file.
> - Run a web server that serves up this metadata file
> - Configure the FileBackedHTTPMetadataProvider in relying-party.xml to
> download this changed file every minute.
> - Create a dummy attribute (or just use an existing one) in LDAP
> - Edit attribute-filter.xml and add 5 to 10
> <afp:AttributeFilterPolicy><afp:PolicyRequirementRule/><afp:AttributeRule/></afp:AttributeFilterPolicy>
> sections that reference the dummy attribute and for each one, provide a
> different matching value. (Example below.)
> - Then test by setting the dummy attribute value to, say, 1 and clearing
> all cookies, and logging into a test SP. (Also, of course, release the
> attribute to the SP that you have in afp:AttributeRule.)
> - Allow the FileBackedHTTPMetadataProvider to download a new version of
> the metadata (it should do this every minute.)
> - Grab the PID of your IDP JVM
> - Do jmap -histo:live <PID> | grep EntitiesDescriptorImpl and note the
> count.
> - Then repeat the above 4 steps, except increment your dummy value to be
> something different now so that a new <afp:AttributeFilterPolicy> is
> triggered (matched) on your next login.
>
> As you repeat those steps, it accumulates cached
> ShibbolethFilteringContexts and EntitiesDescriptorImpl objects in the heap.
>
> Once I was able to duplicate the memory leak in test like this, it became
> easy to test with a different JVM.
>
> So I tested with Oracle JDK 1.7.0_67 Linux x64 instead of the OpenJDK that
> ships with RedHat 6.5.
>
> The jmap -histo:live shows that the EntitiesDescriptorImpl count does
> *not* grow with this JVM (and no stale ShibbolethFilteringContexts).
>
> Now I'm ready to blame OpenJDK. :)
>
> I'll still need to verify by deploying a new Oracle JVM to production and
> confirming the memory leak is gone, but based on the above testing, it
> looks likely that switching to Oracle JDK and avoiding the JVM that ships
> with RedHat will solve the issue.
>
> Too bad because it would be nice to use the package management system on
> RedHat for software updates like the JVM. That was one of the reasons why
> we migrated to RedHat in the first place.
>
> A couple side notes:
>
> - I found a few things on the web that suggests using
> scriptContext.setOptimizationLevel(-1) will prevent Rhino from using the
> ClassCache. (Perhaps the Oracle JDK has this nonoptimization level by
> default while OpenJDK simply has it set to optimize by default?)
> - One potential fix to try for OpenJDK deployments is add
> scriptContext.setOptimizationLevel(-1) to the Shibboleth code to see if it
> avoids the ClassCaching. And if so, I wonder if it's worth considering
> adding to the code. (Is this nonoptimization level the default in Oracle
> JDK?)
> - I saw postings on the web that indicate Rhino has been removed from JDK
> 8 in favor of a "Nashorn." I wonder what the implications of this are for
> anybody who is migrating shibboleth to Java 8 and who has existing
> Javascript scripts in their Shibboleth configs?
>
> Example attribute-filter.xml config using a dummy attribute with
> incrementing values:
>
> <!-- run the filtering script for givenName if MatchingAttribute==1
> -->
> <afp:AttributeFilterPolicy>
> <afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"
> attributeID="MatchingAttribute"
> value="1"/>
> <afp:AttributeRule attributeID="givenName">
> <afp:DenyValueRule xsi:type="basic:Script">
> <basic:Script>
> <![CDATA[
> <SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS>
> ]]>
> </basic:Script>
> </afp:DenyValueRule>
> </afp:AttributeRule>
> </afp:AttributeFilterPolicy>
>
> <!-- run the filtering script for givenName if MatchingAttribute==2
> -->
> <afp:AttributeFilterPolicy>
> <afp:PolicyRequirementRule xsi:type="basic:AttributeValueString"
> attributeID="MatchingAttribute"
> value="2"/>
> <afp:AttributeRule attributeID="givenName">
> <afp:DenyValueRule xsi:type="basic:Script">
> <basic:Script>
> <![CDATA[
> <SOME JAVASCRIPT HERE, DOESN'T MATTER WHAT IT IS>
> ]]>
> </basic:Script>
> </afp:DenyValueRule>
> </afp:AttributeRule>
> </afp:AttributeFilterPolicy>
>
> --
> To unsubscribe from this list send an email to
> users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to
> users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to
> users-unsubscribe at shibboleth.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20141001/804d38cf/attachment-0001.html
More information about the users
mailing list