Problem with tomcat hanging on Shib 2.4

[Matthew Slowe]

On Thu, Aug 28, 2014 at 11:49:11PM +0000, John Kamminga wrote:
>    We've migrated our production Shibboleth environment from Solaris 10 to
>    Redhat 6 and are now experiencing problems with the app becoming
>    unresponsive every couple weeks. A tomcat reboot fixes it, but we'd like
>    to find out what is causing it. Has anyone else experienced issue
>    migrating to or running on Redhat 6?
> 
>    Or, does anyone see any potential problems with our setup?
> 
>    Here is our environment setup on a Redhat VM.
> 
>    Redhat Linux version: 2.6.32-431.20.3.el6.x86_64
> 
>    Shibboleth Idp 2.4
> 
>    Tomcat  6.0.24
> 
>    JAVA_OPTS=" -Xmx1024M -XX:MaxPermSize=512M -server
>    -Djava.library.path=/usr/lib64 -Djavax.net.ssl.trustStore=/jdk/cacerts"
> 
>    Java -version:
>    java version "1.7.0_55"
>    OpenJDK Runtime Environment (rhel-2.4.7.1.el6_5-x86_64 u55-b13)
>    OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

First I'm going to refer to a thread on the JISC-SHIBBOLETH mailing list
last year on the subject (no signin required)

https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=ind1310&L=jisc-shibboleth&F=&S=&P=60

We have three IDPs running in a very similar setup to yourselves (3 RHEL
VMs (each are 2cpu, 4G) on VMware) running, then, 1.7.0_25 (now _55)
each servicing up to 330,000 authentications per day.

Anywhere from a few days to a week or two after startup, the JVM will go
into a wierd state and stop responding to practically anything. It
appears to get stuck doing some massive Garbage Collect which we've not
been able to tune out (which is what that thread is about).

Having sunk days of time into it, we bailed and scheduled rolling
overnight tomcat restarts :-(

Take a look at the GC logs (which you may need to turn on) to see if
you're hitting long GCs (hint, not recommendation):

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
-XX:+PrintGCApplicationStoppedTime -verbose:gc
-Xloggc:/var/log/tomcat6/gc.log

Good luck!
-- 
Matthew Slowe
Server Infrastructure Team	e: m.slowe at kent.ac.uk
IS, University of Kent		t: +44 (0)1227 824265
Canterbury, UK			w: www.kent.ac.uk